Past exam of the advanced topics in statistical machine learning course of the University of Oxford / 2025 / 1 / b

Put $r(\theta )=p(\mathcal{D}\mid \theta )p(\theta )$ and $Z=\int r(\theta )d\theta$. The support condition for importance sampling is essential: $q(\theta )>0$ almost everywhere that $r(\theta )>0$. For one draw $\theta \sim q$, the importance weight $W=r(\theta )/q(\theta )$ satisfiesBy linearity of expectation,This proves that it is an unbiased estimator. If the proposal omits a region of positive target mass, the expected value instead misses that region's contribution; the unconditional unbiasedness claim needs the stated support condition for importance sampling.

With $r$, $Z$, and $W$ as above, independence of the draws gives the variance of a sample mean:This formula allows infinite variance. To identify the optimum, write $\pi (\theta )=p(\theta \mid \mathcal{D})=r(\theta )/Z$. ThenThe last equality uses $\int \pi =\int q=1$. Its nonnegative integrand provesIndeed, all importance weights then equal $Z$. This is a formal optimum: evaluating the normalized posterior density generally already requires the unknown marginal likelihood.

Poor coverage of regions with substantial posterior mass produces large weights. In particular, excessively light proposal tails, a proposal that is too narrow, or a missed mode can make $\int r^2/q$ enormous or infinite. A proposal that is unnecessarily diffuse can also waste nearly all draws on negligible posterior probability. Omitting target support altogether violates unbiasedness, rather than merely increasing variance.

For reciprocal importance sampling, use the same $r$ and $Z$ but draw from the posterior density $\pi =r/Z$. Here the support requirement is reversed: the normalized density $q$ must vanish almost everywhere that $r=0$. Thus $q$ may be confined to a subset of the posterior distribution's support. For $V=q(\theta )/r(\theta )$,Consequently,It is the estimate of the reciprocal that is an unbiased estimator. Taking its reciprocal does not generally produce an unbiased estimator of $Z$: inversion is nonlinear. In fact, when the average is strictly positive and the relevant expectation exists, Jensen's inequality gives $\mathbb{E}{\hat{p}}_2\ge Z$.

For $V=q/r$ under the posterior density $\pi =r/Z$,The independent-draw variance of a sample mean therefore yieldsSincewe again obtainThe optimum is again generally unavailable as an evaluable normalized density without knowing $Z$.

A proposal with too much mass where the posterior density is tiny makes reciprocal weights unstable. In particular, proposal tails that are too heavy relative to the posterior distribution can make $\int q^2/\pi$ infinite. This contrasts with ordinary importance sampling, whose dangerous denominator is $q$. Choosing a normalized $q$ restricted to a well-covered posterior region is allowed, although a very small region can make most draws contribute zero and increase variance. Taking $q=p(\theta )$ gives the Harmonic mean estimator of Bayesian model evidence, illustrating why very small likelihood values can cause severe instability.