Past exam of the mathematics course of the University of Cambridge / 2007 / iii / Paper 48 / 6 / b

Introduce a categorical latent variable $Z_i$ for each observation, with $P(Z_i=j)={\alpha }_j$, and let $z_{ij}=1_{\{Z_i=j\}}$. Conditional on $Z_i=j$, $X_i$ has a normal distribution of mean ${\mu }_j$ and the shared variance ${\sigma }^2$. The complete-data log-likelihood function isInitialize finite means, positive weights summing to one, and a positive common variance. At iteration $r$, the Bayes' theorem gives the E-step mixture responsibilitiesThe common normal-density factor cancels. The observations' conditional labels are independent, and replacing $z_{ij}$ by their conditional expectations $r_{ij}$ givesSet $n_j={\sum }_i{r}_{ij}$; because ${\sum }_j{r}_{ij}=1$, ${\sum }_j{n}_j=N$. The constrained weight maximization uses a Lagrange multiplier $\eta$: differentiating ${\sum }_j{n}_j\log {\alpha }_j-\eta ({\sum }_j{\alpha }_j-1)$ gives ${\alpha }_j=n_j/\eta$, and summing fixes $\eta =N$. Differentiating the residual quadratic in each mean gives ${\sum }_i{r}_{ij}(x_i-{\mu }_j)=0$. Hence the first two M-step updates areWith these new means define $R={\sum }_{i,j}{r}_{ij}(x_i-{\mu }_j^{(r+1)}{)}^2$. The remaining objective in $u={\sigma }^2$ is $-N\log u/2-R/(2u)$, whose derivative is $-N/(2u)+R/(2u^2)$. For $R>0$ it increases up to $u=R/N$ and decreases afterwards. Thus the shared-variance update isThe denominator is $N$, not $N-k$ and not a separate count for each component: this is maximum likelihood estimation with one common variance. These are the EM for Gaussian mixtures with a common variance updates. Recompute responsibilities with the new parameters and repeat until the observed-data likelihood changes sufficiently little.

The monotonicity claim also follows directly. For positive old weights, the likelihood ratio for observation $i$ isApply the Jensen inequality to the logarithm and sum over observations. This yieldsfor the maximizing M step. This proves likelihood increase, not convergence to the global maximum. Use several initializations to reduce sensitivity to local maxima. Compute responsibilities with a log-sum-exp function normalization to avoid numerical underflow.

If $n_j=0$, that component's updated weight is zero and its mean is unidentified; retain its old mean or remove the empty component. Strictly positive interior initialization avoids this case in exact arithmetic while the variance stays positive. If $R=0$, there is no positive finite variance maximizing this M step: the objective tends to infinity as ${\sigma }^2\downarrow 0$. Such boundary degeneracy needs a variance constraint or regularization, as discussed in the next solution. For $k=1$ the formulas reduce to the ordinary sample mean and the maximum-likelihood variance $N^{-1}{\sum }_i(x_i-\overline{x}{)}^2$.

Fit the finite Gaussian mixture with a common variance for candidate values $k=1,\dots ,K$, using several initializations of the expectation-maximization algorithm for each, and retain the best likelihood value found. Maximized likelihood alone cannot select the number of components: a smaller mixture embeds in a larger one by giving an added component zero weight, so its likelihood cannot decrease.

There are $k-1$ free mixture weights, $k$ means and one common variance, henceA standard working choice is the Bayesian information criterion:The penalty balances fit against parameter count. The Akaike information criterion instead uses $-2\ell ({\hat{\theta }}_k)+4k$ and is another possible comparison, particularly when predictive fit is the goal. The parameter count would differ if component variances were separate, but that is not this model.

Likelihood fitting must first be made well posed. Even a shared variance does not prevent all degeneracy: if $k=N$, choose ${\mu }_j=x_j$, ${\alpha }_j=1/N$ and let the common variance tend to zero. Each observed density is at least $1/(N\sqrt{2\pi {\sigma }^2})$, so the full log-likelihood function tends to $+\infty$. The same construction works when $k$ reaches the number of distinct observed values. Thus one must not feed infinite fitted likelihoods into an information criterion. Restrict candidate sizes to a sensible range below that degeneracy, or impose a fixed positive variance floor and use that same constrained model in every comparison. Under a variance floor $\epsilon$, the M-step variance becomes $\max \{\epsilon ,R/N\}$.

There is also nonregular mixture model selection: at a missing component its weight is on a boundary and its mean is unidentified, while permuting labels leaves the distribution unchanged. Consequently an ordinary chi-squared calibration for a likelihood-ratio test between successive component counts is not generally justified. Information criteria are practical comparisons, not an automatic universal proof that the true number has been recovered. Predictive cross-validation or a suitably constructed parametric bootstrap comparison can supplement them. Select the component count by a penalized, well-defined mixture fit, rather than by unpenalized likelihood increase.