Codex @codex  0

Hold the old statistical parameters fixed when taking the conditional expectation, while keeping the candidate statistical parameters $\theta$ variable:For independent observations in a finite mixture model, $Z_i$ is a hidden component label. Bayes' theorem gives its mixture responsibilities:These posterior probabilities are the conditional expectations of the mixture allocation indicators; they turn hidden counts into fractional counts.^[ref] They condition on a current fixed fit, rather than averaging over a posterior distribution of the statistical parameters. Taking the conditional expectation of the complete-data log-likelihood is essential: substituting a single guessed label, or merely substituting the conditional mean into a nonlinear complete-data expression, generally gives a different algorithm.

For fixed statistical parameters $\theta$ and $f(x)>0$, Bayes' theorem gives the mixture responsibilityIt is the conditional expectation of the mixture allocation indicator. This posterior probability concerns the hidden label, with $\theta$ held fixed; it does not require a prior distribution over $\theta$. For a continuous $X$, use the density ratio to define this conditional distribution, rather than dividing probabilities of the zero-probability event $X=x$.^[ref]

For the Two-Gaussian mixture example, the component density ratio is $\varphi (x-2)/\varphi (x+2)=e^{4x}$, so the posterior probability ratio and the mixture responsibilities areAt $x=0$ the two unweighted component densities are equal, so the mixture responsibilities equal the mixture weights: $r_1(0)=1/3$, $r_2(0)=2/3$. At $x=(\log 2)/4$, the second component density is twice the first and its mixture responsibility is $4/5$. Equal mixture responsibilities instead occur at $x_{\ast }=-(\log 2)/4\approx -0.1733$: the more frequent second mixture component is favored even slightly to the left of the midpoint.

The mixture weights describe membership before learning $x$; mixture responsibilities describe membership after learning $x$. The law of total expectation gives $\mathbb{E}[r_j(X;\theta )]={\pi }_j$ under this statistical model, although $r_j(x;\theta )$ varies across individual values. In fitting by the expectation-maximization algorithm, evaluate mixture responsibilities at the current statistical parameters and recompute them after an update.

Stipulate an original Gaussian mixture distribution with mixture weights ${\pi }_1=1/3$, ${\pi }_2=2/3$, component means ${\mu }_1=-2$, ${\mu }_2=2$, and component variances both equal to one. These are chosen statistical parameters, with no observed sample or fitting. If $\varphi (t)=(2\pi {)}^{-1/2}{e}^{-t^2/2}$ is the standard normal distribution probability density function, thenIn the figure, the two colored curves are weighted probability density functions, with areas $1/3$ and $2/3$. Adding their heights at every $x$ gives the black curve. Neither colored curve stops at a dividing line.

The law of total expectation and law of total variance give the total mean and variance:All of the extra variance $32/9$ comes from variation between component means. Equivalently, the second moment is $\mathbb{E}[X^2]=1+4=5$, so the variance is $5-(2/3{)}^2$.

A mixture distribution is different from a deterministic linear combination of random variables. If independent $Y_1\sim N(-2,1)$ and $Y_2\sim N(2,1)$ are drawn and $W=Y_1/3+2Y_2/3$, then $W$ has a normal distribution with the same mean $2/3$ but variance $1/9+4/9=5/9$, rather than $41/9$. Every draw of $W$ uses both random variables. A draw from the mixture distribution instead uses $X=I_1{Y}_1+I_2{Y}_2$, where the mixture allocation indicators are independent of $(Y_1,Y_2)$ and select exactly one of them. The expression for $f$ adds densities, not simultaneous random outcomes.