Past exam of the mathematics course of the University of Cambridge / 2007 / iii / Paper 48 / 6 / b / i / Solution

Introduce a categorical latent variable $Z_i$ for each observation, with $P(Z_i=j)={\alpha }_j$, and let $z_{ij}=1_{\{Z_i=j\}}$. Conditional on $Z_i=j$, $X_i$ has a normal distribution of mean ${\mu }_j$ and the shared variance ${\sigma }^2$. The complete-data log-likelihood function isInitialize finite means, positive weights summing to one, and a positive common variance. At iteration $r$, the Bayes' theorem gives the E-step mixture responsibilitiesThe common normal-density factor cancels. The observations' conditional labels are independent, and replacing $z_{ij}$ by their conditional expectations $r_{ij}$ givesSet $n_j={\sum }_i{r}_{ij}$; because ${\sum }_j{r}_{ij}=1$, ${\sum }_j{n}_j=N$. The constrained weight maximization uses a Lagrange multiplier $\eta$: differentiating ${\sum }_j{n}_j\log {\alpha }_j-\eta ({\sum }_j{\alpha }_j-1)$ gives ${\alpha }_j=n_j/\eta$, and summing fixes $\eta =N$. Differentiating the residual quadratic in each mean gives ${\sum }_i{r}_{ij}(x_i-{\mu }_j)=0$. Hence the first two M-step updates areWith these new means define $R={\sum }_{i,j}{r}_{ij}(x_i-{\mu }_j^{(r+1)}{)}^2$. The remaining objective in $u={\sigma }^2$ is $-N\log u/2-R/(2u)$, whose derivative is $-N/(2u)+R/(2u^2)$. For $R>0$ it increases up to $u=R/N$ and decreases afterwards. Thus the shared-variance update isThe denominator is $N$, not $N-k$ and not a separate count for each component: this is maximum likelihood estimation with one common variance. These are the EM for Gaussian mixtures with a common variance updates. Recompute responsibilities with the new parameters and repeat until the observed-data likelihood changes sufficiently little.

The monotonicity claim also follows directly. For positive old weights, the likelihood ratio for observation $i$ isApply the Jensen inequality to the logarithm and sum over observations. This yieldsfor the maximizing M step. This proves likelihood increase, not convergence to the global maximum. Use several initializations to reduce sensitivity to local maxima. Compute responsibilities with a log-sum-exp function normalization to avoid numerical underflow.

If $n_j=0$, that component's updated weight is zero and its mean is unidentified; retain its old mean or remove the empty component. Strictly positive interior initialization avoids this case in exact arithmetic while the variance stays positive. If $R=0$, there is no positive finite variance maximizing this M step: the objective tends to infinity as ${\sigma }^2\downarrow 0$. Such boundary degeneracy needs a variance constraint or regularization, as discussed in the next solution. For $k=1$ the formulas reduce to the ordinary sample mean and the maximum-likelihood variance $N^{-1}{\sum }_i(x_i-\overline{x}{)}^2$.