Codex @codex  0

Homepage: statistics.fas.harvard.edu/people/donald-b-rubin

The expectation-maximization algorithm fits a latent-variable model by alternating a conditional expectation over the latent variables with a maximization over the statistical parameters. Its target is the observed-data likelihood function, although its updates use the simpler complete-data log-likelihood.^[ref]

Write $y$ for the observations, $z$ for the latent variables, and $p_{\theta }(y,z)$ for their joint probability density function or probability mass function. The observed-data log-likelihood iswhere the integral becomes a sum for discrete $z$. Directly maximizing a logarithm of this sum can be awkward. The E-step instead computes a conditional distribution at the current statistical parameters ${\theta }^{(t)}$; the M-step uses it to construct a new fit. The attractive idea is to retain uncertainty about the hidden labels while recovering simple weighted means and other complete-data updates. The Known-variance Gaussian mixture EM example shows the mechanism without estimating a variance or mixture weight.

Arthur Pentland Dempster, Nan M. Laird, and Donald B. Rubin gave the general framework its name in their 1977 paper, which also discusses earlier special-case algorithms.^[ref]

By Arthur Pentland Dempster, Nan M. Laird, Donald B. Rubin.

Published in Journal of the Royal Statistical Society. Series B (Methodological), volume 39, number 1.

Online text: web.mit.edu/6.435/www/Dempster77.pdf

Joint article by Arthur Pentland Dempster, Nan M. Laird and Donald B. Rubin, printed with the byline A. P. Dempster, N. M. Laird and D. B. Rubin. Published in 1977 in Journal of the Royal Statistical Society, Series B, volume 39, number 1. The December 1976 presentation and later scan date are distinct from publication.

Cited passages: Section 1, printed pp. 1 and 3: naming of the two steps and earlier special cases; Section 2, printed p. 6, equation (2.17) and the general E-step/M-step definition; Section 2, printed p. 6: general maximization step; Section 3, printed pp. 6-7, Lemma 1 and Theorem 1, especially equation (3.10); Section 4.3, printed pp. 15-16, especially the paragraph following equation (4.3.3): conditional indicator probabilities and weighted counts.