Past exam of the mathematics course of the University of Cambridge / 2002 / iii / Paper 40 / 4 / b

For observed data $x$ and latent variables $Z$, let ${\ell }_c(\theta ;x,Z)=\log p_{\theta }(x,Z)$ be the complete-data log-likelihood. From a current parameter ${\theta }^{(m)}$, the expectation-maximization algorithm computesand then chooses ${\theta }^{(m+1)}$ maximizing this function in its first argument. Thus the E-step averages the missing information conditional on the current model and observed data; the M-step treats those expectations as fixed while maximizing.

For a short proof of likelihood function monotonicity, let $q_m(Z)=p_{{\theta }^{(m)}}(Z\mid x)$. ThenThe Jensen inequality for the logarithm gives $\ell (\theta ;x)-\ell ({\theta }^{(m)};x)\ge Q(\theta \mid {\theta }^{(m)})-Q({\theta }^{(m)}\mid {\theta }^{(m)})$. The M-step makes the right side nonnegative. Thus observed likelihood function cannot decrease. This proves the usual EM likelihood monotonicity, while allowing convergence to local stationary points rather than guaranteeing a global maximum.

Put ${\alpha }_1=\alpha$, ${\alpha }_2=1-\alpha$, and introduce a categorical class label for each observation, encoded by $Z_{i1},Z_{i2}\in \{0,1\}$ with $Z_{i1}+Z_{i2}=1$. Its prior class probability is ${\alpha }_j$, and conditional on class $j$ the observation probability density function is $f_j(x_i)$. Thus its joint observation-label probability density function isIndependence across observations yields the finite Gaussian mixture with a common variance complete-data likelihood functionSumming over the two possible labels at each observation recovers ${\prod }_i[\alpha f_1(x_i)+(1-\alpha )f_2(x_i)]$, the observed-data likelihood function. The product requested in the PDF is therefore a joint complete-data likelihood function. Strictly conditioning on $Z$ would remove the factors ${\alpha }_j$; the PDF's conditional notation for this joint product is imprecise.

The E-step uses the posterior class probabilitiesLet $n_j^{(m)}={\sum }_i{w}_{ij}^{(m)}$, so $n_1^{(m)}+n_2^{(m)}=n$. Up to constants, the expected complete-data log-likelihood isMaximizing the first term with ${\alpha }_1+{\alpha }_2=1$ gives ${\alpha }^{(m+1)}=n_1^{(m)}/n$. Differentiating in each ${\mu }_j$ gives ${\mu }_j^{(m+1)}={\sum }_i{w}_{ij}^{(m)}{x}_i/n_j^{(m)}$. After these updates, maximizing in the shared variance givesThe residuals use the new component means, while the weights remain from the old parameters. The divisor is $n$, not $n-2$, because this is a maximum-likelihood update. Starting with $0<\alpha <1$ and $v>0$ gives positive responsibilities and positive $n_j$; a zero effective component size at a boundary would leave its mean unidentified. These are the EM for Gaussian mixtures with a common variance transitions.