= Identical-mean EM trap
{c}
In the <Known-variance Gaussian mixture EM example>, initialize both <means> at the same value $m$. The <mixture responsibilities> are then $1/2$ for every observation. Each <M-step> sets both <means> to the sample <mean>, zero here; the next <E-step> again gives $1/2$. Thus $(0,0)$ is a <fixed point> with <log-likelihood> $-14.593631199$, despite the better fit found from $(-1,1)$.
One can check the failure without trusting an iteration plot. Separate the <means> as $(-a,a)$ and use the <hyperbolic cosine> identity:
$$
\ell(-a,a)-\ell(0,0)=-3a^2+\sum_{i=1}^6\log\cosh(ay_i)=\frac{\sum_iy_i^2-6}{2}a^2+O(a^4)=6.08a^2+O(a^4).
$$
The <Taylor expansion> uses $\sum_iy_i^2=18.16$. Small nonzero separation therefore improves the <log-likelihood>, whereas translating both <means> to $m$ changes it by $-3m^2$. This <stationary point> has both improving and worsening directions. Identical initialization cannot discover the separation because every observation keeps identical label probabilities. Trying several distinct initial fits and comparing their final <log-likelihoods> is useful, but does not prove global optimality.
Keeping the <variance> fixed also removes a different pathology. With known $\sigma=1$, every component <probability density function> is at most $(2\pi)^{-1/2}$, so $\ell\leq-3\log(2\pi)$. If separate component <variances> are allowed to shrink without restriction, one <mixture component> can concentrate at an observation while another retains positive density at the other observations, making the <likelihood function> unbounded. The fixed positive <variance> in this example prevents that collapse; it does not remove initialization traps.
Back to article page
