Past exam of the probability, statistics and options I course of the University of Oxford / 2008 / S / 1

Assume the necessary support condition for importance sampling, $p\ll q$; the source's density assumptions alone do not ensure it. Put $\mu =E_{p}f$ and $w=p/q=(Z_q/Z_p)\tilde{p}/\tilde{q}$. The importance sampling estimator isIntegration under $q$ gives $E_q[wf]=\int pf=\mu$, proving unbiasedness. Independence givesThis variance is finite precisely when the first integral is finite. The assumption $E_p{f}^2<\infty$ ensures target square integrability, but need not ensure square integrability after importance reweighting; the variance can be infinite.

For unknown normalization constants, use self-normalized importance sampling with $\tilde{w}=\tilde{p}/\tilde{q}$:Under the same support condition, $E_q\tilde{w}=Z_p/Z_q>0$ and $E_q[\tilde{w}f]=(Z_p/Z_q)\mu$. Moreover $E_q|\tilde{w}f|=(Z_p/Z_q)E_p|f|<\infty$, by Cauchy-Schwarz inequality and the supplied $E_p{f}^2<\infty$. The strong law of large numbers applies to numerator and denominator sample averages. Their ratio tends almost surely, hence in probability, to $\mu$. If a finite sample happens to give zero denominator, it can be assigned an arbitrary value; this does not affect the eventual almost sure limit. The ratio estimator is generally biased at finite $N$.

The importance weight for the two normal densities isUse ${\hat{\mu }}_N=N^{-1}{\sum }_i{w}_{\sigma }(X_i)f(X_i)$ with $X_i\sim N(0,1)$. Its second moment before averaging isFor the source's general assumption $E_p{f}^2<\infty$, a guaranteed range isIndeed $w_{\sigma }\le 1/\sigma$ there, so $E_q(w_{\sigma }^2{f}^2)=E_p(w_{\sigma }{f}^2)\le {\sigma }^{-1}{E}_p{f}^2<\infty$.

This range cannot be enlarged uniformly over all such $f$. If $\sigma >1$, choose $f^2=e^{a{x}^2}$ withThen $E_p{f}^2<\infty$ but the displayed weighted second-moment integral diverges. Thus no variance range depending only on $\sigma$ can characterize every individual integrand beyond the guaranteed range.

If an additional boundedness assumption on $f$ is intended, the familiar sufficient range becomes $0<\sigma <\sqrt{2}$. For $f=1$ this condition is also necessary, but a compactly supported $f$ can have finite weighted variance for every $\sigma$. Distinguishing these assumptions prevents imposing an unstated restriction on the source integrand.

For the shifted normal proposal, $w(x)=e^{-\mu x+{\mu }^2/2}$ and the estimator isLet $\bar{\Phi }=1-\Phi$. Completing the square gives its unaveraged second momentThe variance is $[I(\mu )-\bar{\Phi }(z{)}^2]/N$, so only $I$ needs minimization. It tends to infinity at both ends of the real $\mu$ axis and has a finite minimizer. Differentiation at a minimizer yieldsFor a standard normal variable, integration of $x\phi (x)=-{\phi }^{\prime }(x)$ shows $E_p[X\mid X>a]=\phi (a)/\bar{\Phi }(a)$. ConsequentlyThe shift is chosen to reduce the second moment of the weighted rare-event indicator, rather than merely to increase the proposal's tail probability.