Past exam of the mathematics course of the University of Cambridge / 2015 / iii / Paper 33 / 6 / a

For a finite Gaussian mixture with a common variance, introduce independent latent labels $Z_i\in \{1,\dots ,k\}$ with probabilities ${\pi }_j>0$, ${\sum }_j{\pi }_j=1$, and conditional responses $Y_i\mid Z_i=j\sim N({\mu }_j,{\sigma }^2)$, with common ${\sigma }^2>0$. The observed density and log-likelihood areThe expectation-maximization algorithm replaces the difficult log of sums by an expected complete-data objective. Choose positive initial weights and variance and separated initial means. At iteration $r$, the E-step computes the mixture responsibilitiesThe common normalizing factor cancels. For numerical stability the probabilities can be evaluated by subtracting the largest log weight before exponentiating.

With the old responsibilities held fixed, the expected complete-data log-likelihood, up to irrelevant constants, isLet $N_j={\sum }_i{\tau }_{ij}^{(r)}$. A Lagrange multiplier for ${\sum }_j{\pi }_j=1$, weighted least squares for the means, and differentiation in the common variance give the M-step:The variance update uses the new means and old responsibilities, with denominator $n$, not a residual degrees-of-freedom adjustment: this is maximum likelihood estimation. Repeat the E- and M-steps until the observed log-likelihood and parameters stabilize.

By EM likelihood monotonicity, exact updates do not decrease the observed likelihood. They need not reach its global maximum, so use several starting configurations and keep the best converged fit, checking for empty or nearly empty components and vanishing variance. The labels are interchangeable; sorting means after fitting supplies an interpretable labeling. Distinct starting means do not guarantee that all fitted components remain distinct. With a common variance and the usual fixed small $k$ relative to distinct observations the model avoids the individual-component variance-collapse pathology of unrestricted Gaussian mixtures, but degenerate data or too many components still require attention.