Past exam of the mathematics course of the University of Cambridge / 2002 / iii / Paper 40 / 2 / a

Assume independent samples $X_i\sim g$, $\int f|\theta |<\infty$, and $g>0$ wherever $f\theta$ can contribute. The importance weight is $w(x)=f(x)/g(x)$, and the ordinary importance sampling estimator isIts expectation is $\int \theta (x)f(x)dx=\mu$, by cancellation of $g$ in the expectation integral. The strong law of large numbers gives almost-sure convergence to $\mu$. Common probability support is sufficient; the precise support condition for importance sampling only requires covering the nonzero integrand for this particular expectation.

Write $f=\tilde{f}/Z_f$ and $g=\tilde{g}/Z_g$. If the constants needed to evaluate $f/g$ are unavailable, use the self-normalized importance sampling estimatorThe unknown common factor $Z_g/Z_f$ cancels. Under full target probability support and integrability, the numerator and denominator sample means converge to $\int \tilde{f}\theta /Z_g$ and $\int \tilde{f}/Z_g$, respectively, whose ratio is $\mu$. The same construction applies if only one normalization is unknown by taking the other probability density function already normalized. Sampling from $g$ itself is assumed available, even when its normalization is not explicitly known.

This ratio estimator is generally biased at finite $n$, unlike the ordinary normalized-weight estimator. Also, its variance is not the formula in the next part: that formula concerns the ordinary unbiased estimator. Full probability support for $f$ is needed here because the denominator estimates the entire target normalizing integral, not just the probability support of $f\theta$.

Let $Y_i=\theta (X_i)f(X_i)/g(X_i)$. The independent summands have expectation $\mu$ and second momentWhen this integral is finite, independence givesIf the second-moment integral diverges, the estimator can still be unbiased and consistent, but its variance is infinite; the finite-variance assumption cannot be omitted.

Put $A=\int |\theta (x)|f(x)dx$, with $0<A<\infty$. The Cauchy-Schwarz inequality applied to $|\theta |f/\sqrt{g}$ and $\sqrt{g}$ givesEquality holds exactly when $|\theta |f/\sqrt{g}$ is proportional to $\sqrt{g}$ on the contributing probability support. Therefore the minimum-variance importance distribution isIf $\theta$ has one sign, $A=|\mu |$ and the ideal proposal gives zero variance. Its normalization or exact sampling may nevertheless be impractical. If $A=0$, the integrand vanishes almost everywhere and the estimator is identically zero.

The minimizing probability density function may vanish where $\theta =0$ even though $f>0$ there. This is valid for the ordinary estimator, which needs probability support only for its integrand. If one insists on exactly the same positive probability support as $f$, the bound is instead an infimum whenever this vanishing occurs, approached by $g_{\delta }=(1-\delta )g_0+\delta f$ as $\delta \downarrow 0$: because $g_{\delta }\ge (1-\delta )g_0$, the second moment lies between $A^2$ and $A^2/(1-\delta )$, proving convergence to the bound.