Known-variance Gaussian mixture EM example

Choose six illustrative values $y=(-2.2,-1.8,-1,1,1.8,2.2)$, fixed mixture weights ${\pi }_1={\pi }_2=1/2$, known standard deviation $\sigma =1$, and initial means $({\mu }_1^{(0)},{\mu }_2^{(0)})=(-1,1)$. These are constructed data for a Gaussian mixture distribution, not measurements. With $\varphi$ the standard normal distribution probability density function, the fitted log-likelihood isAt the initial means, the component density ratio is $\varphi (y_i-1)/\varphi (y_i+1)=e^{2y_i}$. The first E-step therefore gives

The paired values $y$ and $-y$ have complementary mixture responsibilities, so $N_1^{(0)}=N_2^{(0)}=3$ exactly. The first M-step givesThe hyperbolic tangent expression follows by pairing the contributions at $y$ and $-y$. Assigning every negative value wholly to the first mixture component and every positive value wholly to the second would instead give means $\pm 5/3$. Those hard assignments discard the overlap: the expectation-maximization algorithm is not the same update as K-means clustering.

The observed-data log-likelihood rises from $-11.420338905$ initially to $-10.368745063$ after one update and $-10.340655142$ after two. Further updates approach means $\pm 1.638023833$, with log-likelihood approximately $-10.340509617$. These values describe this initialization; EM likelihood monotonicity supplies no certificate of a global maximum.

For stable numerical evaluation, put $a_{ij}=\log (1/2)-\frac{1}{2}\log (2\pi )-\frac{1}{2}(y_i-{\mu }_j{)}^2$ and $b_i={\max }_j{a}_{ij}$. The log-sum-exp function givesSubtracting $b_i$ keeps at least one exponential equal to one, avoiding the all-zero sum that direct evaluation of tiny component densities can produce.

In the Known-variance Gaussian mixture EM example, initialize both means at the same value $m$. The mixture responsibilities are then $1/2$ for every observation. Each M-step sets both means to the sample mean, zero here; the next E-step again gives $1/2$. Thus $(0,0)$ is a fixed point with log-likelihood $-14.593631199$, despite the better fit found from $(-1,1)$.

One can check the failure without trusting an iteration plot. Separate the means as $(-a,a)$ and use the hyperbolic cosine identity:The Taylor expansion uses ${\sum }_i{y}_i^2=18.16$. Small nonzero separation therefore improves the log-likelihood, whereas translating both means to $m$ changes it by $-3m^2$. This stationary point has both improving and worsening directions. Identical initialization cannot discover the separation because every observation keeps identical label probabilities. Trying several distinct initial fits and comparing their final log-likelihoods is useful, but does not prove global optimality.

Keeping the variance fixed also removes a different pathology. With known $\sigma =1$, every component probability density function is at most $(2\pi {)}^{-1/2}$, so $\ell \le -3\log (2\pi )$. If separate component variances are allowed to shrink without restriction, one mixture component can concentrate at an observation while another retains positive density at the other observations, making the likelihood function unbounded. The fixed positive variance in this example prevents that collapse; it does not remove initialization traps.