= EM likelihood monotonicity
{c}
For $Q(\theta\mid\theta_0)=\mathbb E_{\theta_0}[\log p_\theta(Y,Z)\mid Y]$, the <Jensen inequality> gives
$$
\log p_\theta(Y)-\log p_{\theta_0}(Y)\geq Q(\theta\mid\theta_0)-Q(\theta_0\mid\theta_0).
$$
Thus any <expectation-maximization algorithm> <M-step> that increases $Q$ cannot decrease the observed-data <likelihood function>.<Maximum Likelihood from Incomplete Data via the EM Algorithm by Arthur Pentland Dempster and others>[Section 3, printed pp. 6-7, Lemma 1 and Theorem 1, especially equation (3.10)]{ref}
For a short proof, assume common support and finite expressions, and keep the <E-step> <conditional distribution> $q_t(z)=p_{\theta^{(t)}}(z\mid y)$ fixed. The <concavity> of the <logarithm> and <Jensen's inequality> give
$$
\ell(\theta)=\log\mathbb E_{q_t}\left[\frac{p_\theta(y,Z)}{q_t(Z)}\right]\geq\mathbb E_{q_t}[\log p_\theta(y,Z)-\log q_t(Z)]=F(q_t,\theta).
$$
This <evidence lower bound> equals $Q(\theta\mid\theta^{(t)})-\mathbb E_{q_t}\log q_t(Z)$. At $\theta=\theta^{(t)}$, the ratio inside the <expectation> is the constant $p_{\theta^{(t)}}(y)$, so equality holds. During the <M-step>, the second term is constant; increasing $Q$ raises a lower bound that touches the current <log-likelihood>. Hence
$$
\ell(\theta^{(t+1)})\geq F(q_t,\theta^{(t+1)})\geq F(q_t,\theta^{(t)})=\ell(\theta^{(t)}).
$$
Increasing <likelihood> values alone do not establish convergence of the <statistical parameters> or attainment of a global maximum. In a differentiable finite <mixture model>, the <gradient> of $\ell$ equals the candidate-parameter <gradient> of $Q$ evaluated at the current fit. Thus an interior <fixed point> of an exact <M-step> is a <stationary point>, which need not be a <local maximum>; the <Identical-mean EM trap> is an explicit counterexample. Small changes in <log-likelihood> can also occur before the <means> have settled, so inspecting both is useful.
Back to article page
