Mixture model ($f(x)={\sum }_{j=1}^K{\pi }_j{f}_j(x)$)

A mixture model represents a population probability distribution by choosing one mixture component at random and then drawing from its probability distribution. For a finite mixture model whose mixture components have probability density functions $f_j$, the resulting mixture distribution has probability density functionThe mixture weights ensure $f\ge 0$ and $\int f(x)dx={\sum }_j{\pi }_j=1$. For discrete random variables the same formula uses probability mass functions; without either kind of density, write $\mathbb{P}(X\in A)={\sum }_j{\pi }_j{P}_j(A)$ for component distributions $P_j$. A zero mixture weight makes that mixture component inactive.^[ref]

The mixture components need not occupy disjoint bins of $x$. Two normal distributions both have positive probability density functions everywhere, so the same value can plausibly come from either mixture component. The Two-Gaussian mixture example makes this overlap visible, and its mixture responsibilities quantify the remaining uncertainty about the latent variable. This is useful when a single normal distribution cannot represent the population's shape, but a fitted mixture model alone does not establish that its mixture components are real biological or social groups.

For a two-component mixture model with known component tail probabilities $p_0,p_1$ above a threshold, the overall probability is $p(q)=(1-q)p_0+q{p}_1$. An empirical exceedance fraction $\hat{p}$ gives the displayed method-of-moments estimate when $p_1\ne p_0$. Under independent observations, the exceedance count has a binomial distribution, and its constrained maximum-likelihood estimate is the same expression clipped to $[0,1]$. Equal component tail probabilities make this summary uninformative about $q$.

A Gaussian scale mixture is a random variable $X=RZ$, where $Z$ has a standard normal distribution and is independent of a positive random scale $R$. Its probability density function is the average of the conditional normal densities. Random scales can produce larger kurtosis than a single normal law.

An independent unit-scale Rayleigh distribution variable $R$ and standard normal distribution variable $Z$ give $RZ$ with Laplace distribution of density $e^{-|x|}/2$. Indeed its characteristic function is $\mathbb{E}{e}^{-t^2{R}^2/2}=1/(1+t^2)$.

A finite mixture model draws each observation from one of finitely many component distributions according to a categorical latent label.

A mixture component is the conditional distribution of $X$ given a component label $Z=j$. Its probability density function $f_j$ integrates to one; its weighted contribution ${\pi }_j{f}_j$ to the mixture distribution integrates to the mixture weight ${\pi }_j$. Thus a mixture component is a distribution, rather than a region assigned to it by a clustering rule.

Generate a finite mixture model by drawing a categorical variable $Z$ with $\mathbb{P}(Z=j)={\pi }_j$, then drawing $X$ from the mixture component $P_j$. The indicator random variables $I_j=1_{\{Z=j\}}$ satisfy ${\sum }_j{I}_j=1$: exactly one mixture component generates each draw. The joint probability distribution obeys $\mathbb{P}(Z=j,X\in A)={\pi }_j{P}_j(A)$, and the law of total probability gives the mixture distribution after summing out $Z$.^[ref]

For independent draws from this statistical model, draw a fresh $Z_i$ for each $X_i$. Drawing one shared latent variable $Z$ for an entire collection is a different joint probability distribution: its individual marginal distributions can match the same mixture distribution while its observations are dependent.

Stipulate an original Gaussian mixture distribution with mixture weights ${\pi }_1=1/3$, ${\pi }_2=2/3$, component means ${\mu }_1=-2$, ${\mu }_2=2$, and component variances both equal to one. These are chosen statistical parameters, with no observed sample or fitting. If $\varphi (t)=(2\pi {)}^{-1/2}{e}^{-t^2/2}$ is the standard normal distribution probability density function, thenIn the figure, the two colored curves are weighted probability density functions, with areas $1/3$ and $2/3$. Adding their heights at every $x$ gives the black curve. Neither colored curve stops at a dividing line.

The law of total expectation and law of total variance give the total mean and variance:All of the extra variance $32/9$ comes from variation between component means. Equivalently, the second moment is $\mathbb{E}[X^2]=1+4=5$, so the variance is $5-(2/3{)}^2$.

A mixture distribution is different from a deterministic linear combination of random variables. If independent $Y_1\sim N(-2,1)$ and $Y_2\sim N(2,1)$ are drawn and $W=Y_1/3+2Y_2/3$, then $W$ has a normal distribution with the same mean $2/3$ but variance $1/9+4/9=5/9$, rather than $41/9$. Every draw of $W$ uses both random variables. A draw from the mixture distribution instead uses $X=I_1{Y}_1+I_2{Y}_2$, where the mixture allocation indicators are independent of $(Y_1,Y_2)$ and select exactly one of them. The expression for $f$ adds densities, not simultaneous random outcomes.

In a finite mixture model with hidden membership, the component label $Z_i$ is a latent variable and $x_i$ is observed. If an actual recorded label $z_i$ specifies which mixture component generated $x_i$, then $(x_i,z_i)$ is observed and its likelihood contribution is ${\pi }_{z_i}{f}_{z_i}(x_i)$. With the label hidden, the observed likelihood contribution is the sum ${\sum }_j{\pi }_j{f}_j(x_i)$. A hard assignment chosen from mixture responsibilities is an inference under the statistical model, not an additional recorded label.

Simultaneously applying a permutation to the mixture weights and mixture components leaves $f$ unchanged. Thus identifiability of named components requires a convention, such as ordering distinct component means; label switching describes the associated inferential issue.^[ref]

Even after choosing label names, identical mixture components reveal only their combined mixture weight, and an inactive mixture component has undetermined statistical parameters. Adding mixture components increases flexibility but can cause overfitting; judge predictions using cross-validation, rather than interpreting every fitted component as a discovered population.^[ref]

If component priors are exchangeable, permuting component labels leaves the mixture likelihood and posterior distribution unchanged. A sampling chain can visit these equivalent modes, making naive component-by-component averages misleading. Predictive densities and other permutation-invariant summaries avoid the issue; component-specific interpretation requires an explicit identification or relabelling rule. Nonexchangeable weight priors can break exact symmetry but do not remove every multimodality problem.

In a finite normal distribution mixture with allocation labels $z_j$, independently assign ${\mu }_i\sim N(m_0,{\tau }^2)$, $v_i\sim \mathrm{InvGamma}(\alpha ,\beta )$ and weights a Dirichlet distribution. Let $n_i$ count allocations to component $i$ and $s_i$ sum their observations. The conditional mean precision is $V_i^{-1}={\tau }^{-2}+n_i/v_i$, giving ${\mu }_i\mid \cdots \sim N(V_i(m_0/{\tau }^2+s_i/v_i),V_i)$. The conditional variance is the displayed inverse-gamma distribution; weights have Dirichlet parameters increased by allocation counts. Allocation probabilities are proportional to ${\omega }_i\phi (x_j;{\mu }_i,v_i)$. Independence of the mean prior from $v_i$ is important: its density does not add an extra inverse-gamma shape term.

A mixture responsibility is the posterior probability of a component label given an observation and current parameters: ${\tau }_{ij}={\pi }_j{f}_j(y_i)/({\sum }_l{\pi }_l{f}_l(y_i))$. Each observation's responsibilities sum to one. The expectation-maximization algorithm uses these soft assignments rather than fixed hard cluster labels.

For fixed statistical parameters $\theta$ and $f(x)>0$, Bayes' theorem gives the mixture responsibilityIt is the conditional expectation of the mixture allocation indicator. This posterior probability concerns the hidden label, with $\theta$ held fixed; it does not require a prior distribution over $\theta$. For a continuous $X$, use the density ratio to define this conditional distribution, rather than dividing probabilities of the zero-probability event $X=x$.^[ref]

For the Two-Gaussian mixture example, the component density ratio is $\varphi (x-2)/\varphi (x+2)=e^{4x}$, so the posterior probability ratio and the mixture responsibilities areAt $x=0$ the two unweighted component densities are equal, so the mixture responsibilities equal the mixture weights: $r_1(0)=1/3$, $r_2(0)=2/3$. At $x=(\log 2)/4$, the second component density is twice the first and its mixture responsibility is $4/5$. Equal mixture responsibilities instead occur at $x_{\ast }=-(\log 2)/4\approx -0.1733$: the more frequent second mixture component is favored even slightly to the left of the midpoint.

The mixture weights describe membership before learning $x$; mixture responsibilities describe membership after learning $x$. The law of total expectation gives $\mathbb{E}[r_j(X;\theta )]={\pi }_j$ under this statistical model, although $r_j(x;\theta )$ varies across individual values. In fitting by the expectation-maximization algorithm, evaluate mixture responsibilities at the current statistical parameters and recompute them after an update.

A finite Gaussian mixture with a common variance has density $f(y)={\sum }_{j=1}^k{\pi }_j\varphi (y;{\mu }_j,{\sigma }^2)$, with positive common variance and weights summing to one. The means locate its latent components, which need not correspond to distinct modes. It has $2k$ free parameters: $k-1$ weights, $k$ means and one variance. EM for Gaussian mixtures with a common variance supplies iterative likelihood updates.

A mixture regression with shared slopes models $Y_i\mid Z_i=j,x_i\sim N({\alpha }_j+x_i^T\beta ,{\sigma }^2)$ with common slopes and component-specific intercepts. It jointly estimates mean adjustment and latent groups, rather than fitting a mixture to fixed residuals. If both a global intercept and component offsets are included, an identifying constraint is required. Constant weights assume latent membership proportions do not depend on the covariates.

Residual mixture clustering first removes a fitted covariate-dependent mean and then explores latent groups in the regression residuals. It seeks differences relative to the adjustment rather than raw-response differences. Mixture responsibilities express uncertain membership. The fitted mixture alone does not establish distinct real-world populations, and two-stage residual mixture fitting ignores some adjustment uncertainty.

Fitting a mixture model to regression residuals treats the fitted mean adjustment as fixed. Even with independent homoscedastic errors, ordinary least-squares residuals have covariance matrix ${\sigma }^2(I-H)$ and satisfy fitted linear constraints, so they are not exactly independent observations. A joint mixture regression with shared slopes or a bootstrap of the whole fitting process better accounts for the first-stage estimation uncertainty.

The E-step computes mixture responsibilities ${\tau }_{ij}$ from the old parameters. Put $N_j={\sum }_i{\tau }_{ij}$. The M-step updates ${\pi }_j=N_j/n$, ${\mu }_j={\sum }_i{\tau }_{ij}{y}_i/N_j$ and ${\sigma }^2={\sum }_{i,j}{\tau }_{ij}(y_i-{\mu }_j^{\mathrm{new}}{)}^2/n$. The variance uses new means and old responsibilities. EM likelihood monotonicity guarantees nondecrease of the observed likelihood, not a global optimum.

A mixture weight is the probability assigned to one component of a mixture model; all mixture weights are nonnegative and sum to one.

A Dirichlet process mixture model uses a Dirichlet process as a random mixing distribution, allowing the number of occupied mixture components to be inferred from the data.