Past exam of the advanced topics in statistical machine learning course of the University of Oxford / 2025 / 1

The marginal likelihood integrates the likelihood function against the prior density:It is the normalizing constant in Bayes' theorem. For the subsequent posterior distribution and importance sampling calculations, assume a normalized prior density and $0<Z<\infty$.

Write $h(\theta )=\log p(\mathcal{D}\mid \theta )+\log p(\theta )$. Choose an interior maximum a posteriori estimate, a globally maximizing posterior modeSuppose $h$ is twice continuously differentiable near $\mu$ and its negative Hessian matrix $A=-{\nabla }^{2}h(\mu )$ is a positive-definite matrix. The second-order Taylor expansion isExponentiating and normalizing this local approximation gives the Laplace approximationThus the approximate mean is the posterior mode, and the approximate covariance matrix is the inverse local curvature. This normal approximation is useful when most posterior probability is concentrated around this mode. A boundary mode, singular curvature, or several substantial modes requires a modified approximation.

Evaluate Bayes' theorem at the posterior mode $\mu$:The multivariate normal density in the Laplace approximation has height $(2\pi {)}^{-d/2}|\Sigma {|}^{-1/2}$ at its mean. ThereforeEquivalently, the approximate log marginal likelihood isThe determinant factor accounts for the volume of the locally plausible parameter region, which the peak likelihood alone does not measure.

Put $r(\theta )=p(\mathcal{D}\mid \theta )p(\theta )$ and $Z=\int r(\theta )d\theta$. The support condition for importance sampling is essential: $q(\theta )>0$ almost everywhere that $r(\theta )>0$. For one draw $\theta \sim q$, the importance weight $W=r(\theta )/q(\theta )$ satisfiesBy linearity of expectation,This proves that it is an unbiased estimator. If the proposal omits a region of positive target mass, the expected value instead misses that region's contribution; the unconditional unbiasedness claim needs the stated support condition for importance sampling.

With $r$, $Z$, and $W$ as above, independence of the draws gives the variance of a sample mean:This formula allows infinite variance. To identify the optimum, write $\pi (\theta )=p(\theta \mid \mathcal{D})=r(\theta )/Z$. ThenThe last equality uses $\int \pi =\int q=1$. Its nonnegative integrand provesIndeed, all importance weights then equal $Z$. This is a formal optimum: evaluating the normalized posterior density generally already requires the unknown marginal likelihood.

Poor coverage of regions with substantial posterior mass produces large weights. In particular, excessively light proposal tails, a proposal that is too narrow, or a missed mode can make $\int r^2/q$ enormous or infinite. A proposal that is unnecessarily diffuse can also waste nearly all draws on negligible posterior probability. Omitting target support altogether violates unbiasedness, rather than merely increasing variance.

For reciprocal importance sampling, use the same $r$ and $Z$ but draw from the posterior density $\pi =r/Z$. Here the support requirement is reversed: the normalized density $q$ must vanish almost everywhere that $r=0$. Thus $q$ may be confined to a subset of the posterior distribution's support. For $V=q(\theta )/r(\theta )$,Consequently,It is the estimate of the reciprocal that is an unbiased estimator. Taking its reciprocal does not generally produce an unbiased estimator of $Z$: inversion is nonlinear. In fact, when the average is strictly positive and the relevant expectation exists, Jensen's inequality gives $\mathbb{E}{\hat{p}}_2\ge Z$.

For $V=q/r$ under the posterior density $\pi =r/Z$,The independent-draw variance of a sample mean therefore yieldsSincewe again obtainThe optimum is again generally unavailable as an evaluable normalized density without knowing $Z$.

A proposal with too much mass where the posterior density is tiny makes reciprocal weights unstable. In particular, proposal tails that are too heavy relative to the posterior distribution can make $\int q^2/\pi$ infinite. This contrasts with ordinary importance sampling, whose dangerous denominator is $q$. Choosing a normalized $q$ restricted to a well-covered posterior region is allowed, although a very small region can make most draws contribute zero and increase variance. Taking $q=p(\theta )$ gives the Harmonic mean estimator of Bayesian model evidence, illustrating why very small likelihood values can cause severe instability.

Use the factorization $p(\theta ,\mathcal{D})=p(\theta \mid \mathcal{D})p(\mathcal{D})$ inside the evidence lower bound:HenceThe marginal likelihood is independent of the variational parameter $\varphi$, so maximizing the evidence lower bound is equivalent to minimizing the Kullback-Leibler divergence. Nonnegativity of the Kullback-Leibler divergence also gives $\mathcal{L}(\varphi )\le \log p(\mathcal{D})$.

The independent coordinates make the logarithm of the probability density ratio a sum. For coordinate $i$, its contribution to the Kullback-Leibler divergence isbecause ${\mathbb{E}}_q{\theta }_i^2={\varphi }^2$. Thus, retaining all constants,Here ${\sigma }_1,{\sigma }_2>0$ and $\varphi >0$, so both normal densities are nondegenerate.

The printed minimization of the evidence lower bound has no solution. To distinguish that literal result from the usual variational inference optimum, put $S={\sigma }_1^{-2}+{\sigma }_2^{-2}>0$. The preceding Kullback-Leibler divergence satisfiesIts unique global minimum occurs at ${\varphi }^2=2/S$. Since $\mathcal{L}=\log p(\mathcal{D})-\mathcal{J}$, this is the unique global maximum of the evidence lower bound:The optimal common variance is the harmonic mean of the two posterior variances. However, $\mathcal{J}(\varphi )\to +\infty$ as either $\varphi \downarrow 0$ or $\varphi \to \infty$. ThereforeThe source PDF confirms the minimization wording; replacing it by maximization would give the finite value above, consistently with part (i).