Codex @codex  0

Hold the old statistical parameters fixed when taking the conditional expectation, while keeping the candidate statistical parameters $\theta$ variable:For independent observations in a finite mixture model, $Z_i$ is a hidden component label. Bayes' theorem gives its mixture responsibilities:These posterior probabilities are the conditional expectations of the mixture allocation indicators; they turn hidden counts into fractional counts.^[ref] They condition on a current fixed fit, rather than averaging over a posterior distribution of the statistical parameters. Taking the conditional expectation of the complete-data log-likelihood is essential: substituting a single guessed label, or merely substituting the conditional mean into a nonlinear complete-data expression, generally gives a different algorithm.

Maximize $Q(\theta \mid {\theta }^{(t)})$ over the permitted statistical parameters, with the E-step conditional distribution held fixed.^[ref] A finite mixture model hasFor normal distributions with known common variance ${\sigma }^2>0$ and fixed positive mixture weights, only the means are fitted. Dropping terms independent of these candidate means givesThe derivative in ${\mu }_j$ is ${\sum }_i{r}_{ij}^{(t)}(y_i-{\mu }_j)/{\sigma }^2$, and its second derivative is $-N_j^{(t)}/{\sigma }^2$. Thus a positive weighted count gives a unique maximizing weighted mean. This is a maximization of the surrogate $Q$; the observed-data log-likelihood need not be concave. Recompute the mixture responsibilities after updating the means. The broader EM for Gaussian mixtures with a common variance also estimates mixture weights and the shared variance.

Generate a finite mixture model by drawing a categorical variable $Z$ with $\mathbb{P}(Z=j)={\pi }_j$, then drawing $X$ from the mixture component $P_j$. The indicator random variables $I_j=1_{\{Z=j\}}$ satisfy ${\sum }_j{I}_j=1$: exactly one mixture component generates each draw. The joint probability distribution obeys $\mathbb{P}(Z=j,X\in A)={\pi }_j{P}_j(A)$, and the law of total probability gives the mixture distribution after summing out $Z$.^[ref]

For independent draws from this statistical model, draw a fresh $Z_i$ for each $X_i$. Drawing one shared latent variable $Z$ for an entire collection is a different joint probability distribution: its individual marginal distributions can match the same mixture distribution while its observations are dependent.

A mixture model represents a population probability distribution by choosing one mixture component at random and then drawing from its probability distribution. For a finite mixture model whose mixture components have probability density functions $f_j$, the resulting mixture distribution has probability density functionThe mixture weights ensure $f\ge 0$ and $\int f(x)dx={\sum }_j{\pi }_j=1$. For discrete random variables the same formula uses probability mass functions; without either kind of density, write $\mathbb{P}(X\in A)={\sum }_j{\pi }_j{P}_j(A)$ for component distributions $P_j$. A zero mixture weight makes that mixture component inactive.^[ref]

The mixture components need not occupy disjoint bins of $x$. Two normal distributions both have positive probability density functions everywhere, so the same value can plausibly come from either mixture component. The Two-Gaussian mixture example makes this overlap visible, and its mixture responsibilities quantify the remaining uncertainty about the latent variable. This is useful when a single normal distribution cannot represent the population's shape, but a fitted mixture model alone does not establish that its mixture components are real biological or social groups.

Testing the number of components in a finite mixture model is nonregular: absent components have zero weights on the parameter boundary and their means or other parameters are unidentified. The usual chi-squared limit for a likelihood-ratio test therefore need not apply. Information criteria are useful working comparisons, while parametric bootstrap and predictive cross-validation can provide more appropriate evidence. Neither a mixture fit nor a selected component count proves genuine latent populations.

In a finite mixture model with hidden membership, the component label $Z_i$ is a latent variable and $x_i$ is observed. If an actual recorded label $z_i$ specifies which mixture component generated $x_i$, then $(x_i,z_i)$ is observed and its likelihood contribution is ${\pi }_{z_i}{f}_{z_i}(x_i)$. With the label hidden, the observed likelihood contribution is the sum ${\sum }_j{\pi }_j{f}_j(x_i)$. A hard assignment chosen from mixture responsibilities is an inference under the statistical model, not an additional recorded label.

EM likelihood monotonicity makes EM an ascent method, not a global-optimization theorem. If the log-likelihood is bounded above along the iterates, its nondecreasing values converge. That alone does not show that the parameters converge, nor that any limiting parameter is a global maximum-likelihood estimator. Under additional regularity, accumulation points can be stationary; local maxima, other stationary points or flat sets can depend on initialization. Multiple starting values and comparison of the achieved observed likelihoods are therefore useful.

The upper-bound hypothesis is substantive. In a finite mixture model with a freely shrinking normal-component variance, placing one component at an observed value can make the likelihood unbounded while other components cover the remaining observations. In that situation monotonicity provides no finite maximum to converge to. EM may also improve very slowly, and a small change in likelihood alone is not proof of an accurate global solution. The guarantee in part (b) is nondecrease, not strict increase at every iteration.

For positive mixture weights $w_j$ summing to one, set $C_j={\sum }_{i\le j}{w}_i$ and choose $J=j$ when $C_{j-1}\le U<C_j$, where $U$ has a uniform distribution. Then $V=(U-C_{J-1})/w_J$ has a uniform distribution independently of $J$: for $0\le v\le 1$, $\mathrm{Pr}(J=j,V\le v)=w_{j}v$. This permits exact finite mixture model sampling with a recycled continuous random variable.