Past exam of the advanced topics in statistical machine learning course of the University of Oxford / 2025 / 1 / a

The marginal likelihood integrates the likelihood function against the prior density:It is the normalizing constant in Bayes' theorem. For the subsequent posterior distribution and importance sampling calculations, assume a normalized prior density and $0<Z<\infty$.

Write $h(\theta )=\log p(\mathcal{D}\mid \theta )+\log p(\theta )$. Choose an interior maximum a posteriori estimate, a globally maximizing posterior modeSuppose $h$ is twice continuously differentiable near $\mu$ and its negative Hessian matrix $A=-{\nabla }^{2}h(\mu )$ is a positive-definite matrix. The second-order Taylor expansion isExponentiating and normalizing this local approximation gives the Laplace approximationThus the approximate mean is the posterior mode, and the approximate covariance matrix is the inverse local curvature. This normal approximation is useful when most posterior probability is concentrated around this mode. A boundary mode, singular curvature, or several substantial modes requires a modified approximation.

Evaluate Bayes' theorem at the posterior mode $\mu$:The multivariate normal density in the Laplace approximation has height $(2\pi {)}^{-d/2}|\Sigma {|}^{-1/2}$ at its mean. ThereforeEquivalently, the approximate log marginal likelihood isThe determinant factor accounts for the volume of the locally plausible parameter region, which the peak likelihood alone does not measure.