Past exam of the advanced topics in statistical machine learning course of the University of Oxford / 2025 / 1 / b / iv / Solution

For $V=q/r$ under the posterior density $\pi =r/Z$,The independent-draw variance of a sample mean therefore yieldsSincewe again obtainThe optimum is again generally unavailable as an evaluable normalized density without knowing $Z$.

A proposal with too much mass where the posterior density is tiny makes reciprocal weights unstable. In particular, proposal tails that are too heavy relative to the posterior distribution can make $\int q^2/\pi$ infinite. This contrasts with ordinary importance sampling, whose dangerous denominator is $q$. Choosing a normalized $q$ restricted to a well-covered posterior region is allowed, although a very small region can make most draws contribute zero and increase variance. Taking $q=p(\theta )$ gives the Harmonic mean estimator of Bayesian model evidence, illustrating why very small likelihood values can cause severe instability.