= Expectation-maximization algorithm
{wiki}
= EM algorithm
{c}
{synonym}
The <expectation-maximization algorithm> fits a <latent-variable model> by alternating a <conditional expectation> over the <latent variables> with a maximization over the <statistical parameters>. Its target is the observed-data <likelihood function>, although its updates use the simpler complete-data <log-likelihood>.<Maximum Likelihood from Incomplete Data via the EM Algorithm by Arthur Pentland Dempster and others>[Section 2, printed p. 6, equation (2.17) and the general E-step/M-step definition]{ref}
Write $y$ for the observations, $z$ for the <latent variables>, and $p_\theta(y,z)$ for their joint <probability density function> or <probability mass function>. The observed-data <log-likelihood> is
$$
\ell(\theta)=\log p_\theta(y),\qquad p_\theta(y)=\int p_\theta(y,z)\,dz,
$$
where the <integral> becomes a sum for discrete $z$. Directly maximizing a <logarithm> of this sum can be awkward. The <E-step> instead computes a <conditional distribution> at the current <statistical parameters> $\theta^{(t)}$; the <M-step> uses it to construct a new fit. The attractive idea is to retain uncertainty about the hidden labels while recovering simple <weighted means> and other complete-data updates. The <Known-variance Gaussian mixture EM example> shows the mechanism without estimating a <variance> or <mixture weight>.
<Arthur Pentland Dempster>, <Nan M. Laird>, and <Donald B. Rubin> gave the general framework its name in their 1977 paper, which also discusses earlier special-case algorithms.<Maximum Likelihood from Incomplete Data via the EM Algorithm by Arthur Pentland Dempster and others>[Section 1, printed pp. 1 and 3: naming of the two steps and earlier special cases]{ref}
Back to article page
