Past exam of the mathematics course of the University of Cambridge / 2007 / iii / Paper 48 / 6 / a / Solution

Both data augmentation and the expectation-maximization algorithm introduce latent variables $Z$ that make the complete-data model easier to handle. The observed-data likelihood is obtained by summing or integrating $p(x,z\mid \theta )$ over the missing data, and both methods use the conditional law of $Z$ given the observations and current parameters.

In data augmentation for Bayesian inference, one alternates random drawsThese are Gibbs sampler updates on the joint posterior distribution; marginal parameter draws explore the whole posterior, including uncertainty, provided the chain converges appropriately. The parameter conditional incorporates its prior.

The expectation-maximization algorithm is instead a deterministic optimization procedure. Its E step formsand its M step maximizes this function over $\theta$. Exact steps do not decrease the observed-data likelihood function. Their aim is a likelihood stationary point, often a local maximum, rather than draws from a parameter distribution. Adding a log prior to the maximization gives a posterior-mode version, which still does not sample the posterior.

Data augmentation samples conditional missing data and parameters; EM averages the complete-data log likelihood and optimizes parameters. The E step is an expectation of the log likelihood, not the log of an averaged likelihood, and generally is not equivalent to merely replacing each missing value by its conditional mean. A Monte Carlo E-step approximation is a further variant of EM, not automatically the same procedure as a data augmentation chain.