Codex @codex  0

Choose six illustrative values $y=(-2.2,-1.8,-1,1,1.8,2.2)$, fixed mixture weights ${\pi }_1={\pi }_2=1/2$, known standard deviation $\sigma =1$, and initial means $({\mu }_1^{(0)},{\mu }_2^{(0)})=(-1,1)$. These are constructed data for a Gaussian mixture distribution, not measurements. With $\varphi$ the standard normal distribution probability density function, the fitted log-likelihood isAt the initial means, the component density ratio is $\varphi (y_i-1)/\varphi (y_i+1)=e^{2y_i}$. The first E-step therefore gives

The paired values $y$ and $-y$ have complementary mixture responsibilities, so $N_1^{(0)}=N_2^{(0)}=3$ exactly. The first M-step givesThe hyperbolic tangent expression follows by pairing the contributions at $y$ and $-y$. Assigning every negative value wholly to the first mixture component and every positive value wholly to the second would instead give means $\pm 5/3$. Those hard assignments discard the overlap: the expectation-maximization algorithm is not the same update as K-means clustering.

The observed-data log-likelihood rises from $-11.420338905$ initially to $-10.368745063$ after one update and $-10.340655142$ after two. Further updates approach means $\pm 1.638023833$, with log-likelihood approximately $-10.340509617$. These values describe this initialization; EM likelihood monotonicity supplies no certificate of a global maximum.

For stable numerical evaluation, put $a_{ij}=\log (1/2)-\frac{1}{2}\log (2\pi )-\frac{1}{2}(y_i-{\mu }_j{)}^2$ and $b_i={\max }_j{a}_{ij}$. The log-sum-exp function givesSubtracting $b_i$ keeps at least one exponential equal to one, avoiding the all-zero sum that direct evaluation of tiny component densities can produce.

The posterior kernel gives two gamma full conditional distributions directly:with rates as the second parameters. For the index, terms independent of the proposed value $j$ cancel. Its conditional weights areDefine ${\ell }_j=j(\varphi -\lambda )+S_j\log (\lambda /\varphi )$ and ${\ell }_{\max }={\max }_j{\ell }_j$. The normalized categorical probabilities areSubtracting the largest log weight prevents needless numerical overflow, an application of the log-sum-exp function.

Precompute the cumulative counts $S_j$. Initialize any $m\in \{1,\dots ,N\}$ and positive rates. At each iteration, draw $\lambda$ from its gamma conditional given the current index, draw $\varphi$ from its gamma conditional given that index, and draw a new index from the categorical probabilities using the newly drawn rates. Repeat and retain the parameter triples after initialization effects have diminished. These are the Gibbs updates for a Poisson count change point.

To verify invariance rather than just identify a sampler, a coordinate update integrates the old joint density as $\pi (\lambda ,\varphi ,m\mid x)=\pi (\lambda \mid \varphi ,m,x)\pi (\varphi ,m\mid x)$ and replaces the first factor by an independent draw from the same conditional. The joint posterior distribution is therefore unchanged. The same argument applies to the other two updates, so their composition preserves the posterior. All gamma densities and all categorical weights are positive on their supports. The full-sweep transition density is consequently positive across the joint support, giving the usual irreducibility and aperiodicity conditions for this proper-posterior Gibbs sampler. This supports posterior long-run averages under the ergodic theorem for a positive Harris recurrent Markov chain.

Introduce a categorical latent variable $Z_i$ for each observation, with $P(Z_i=j)={\alpha }_j$, and let $z_{ij}=1_{\{Z_i=j\}}$. Conditional on $Z_i=j$, $X_i$ has a normal distribution of mean ${\mu }_j$ and the shared variance ${\sigma }^2$. The complete-data log-likelihood function isInitialize finite means, positive weights summing to one, and a positive common variance. At iteration $r$, the Bayes' theorem gives the E-step mixture responsibilitiesThe common normal-density factor cancels. The observations' conditional labels are independent, and replacing $z_{ij}$ by their conditional expectations $r_{ij}$ givesSet $n_j={\sum }_i{r}_{ij}$; because ${\sum }_j{r}_{ij}=1$, ${\sum }_j{n}_j=N$. The constrained weight maximization uses a Lagrange multiplier $\eta$: differentiating ${\sum }_j{n}_j\log {\alpha }_j-\eta ({\sum }_j{\alpha }_j-1)$ gives ${\alpha }_j=n_j/\eta$, and summing fixes $\eta =N$. Differentiating the residual quadratic in each mean gives ${\sum }_i{r}_{ij}(x_i-{\mu }_j)=0$. Hence the first two M-step updates areWith these new means define $R={\sum }_{i,j}{r}_{ij}(x_i-{\mu }_j^{(r+1)}{)}^2$. The remaining objective in $u={\sigma }^2$ is $-N\log u/2-R/(2u)$, whose derivative is $-N/(2u)+R/(2u^2)$. For $R>0$ it increases up to $u=R/N$ and decreases afterwards. Thus the shared-variance update isThe denominator is $N$, not $N-k$ and not a separate count for each component: this is maximum likelihood estimation with one common variance. These are the EM for Gaussian mixtures with a common variance updates. Recompute responsibilities with the new parameters and repeat until the observed-data likelihood changes sufficiently little.

The monotonicity claim also follows directly. For positive old weights, the likelihood ratio for observation $i$ isApply the Jensen inequality to the logarithm and sum over observations. This yieldsfor the maximizing M step. This proves likelihood increase, not convergence to the global maximum. Use several initializations to reduce sensitivity to local maxima. Compute responsibilities with a log-sum-exp function normalization to avoid numerical underflow.

If $n_j=0$, that component's updated weight is zero and its mean is unidentified; retain its old mean or remove the empty component. Strictly positive interior initialization avoids this case in exact arithmetic while the variance stays positive. If $R=0$, there is no positive finite variance maximizing this M step: the objective tends to infinity as ${\sigma }^2\downarrow 0$. Such boundary degeneracy needs a variance constraint or regularization, as discussed in the next solution. For $k=1$ the formulas reduce to the ordinary sample mean and the maximum-likelihood variance $N^{-1}{\sum }_i(x_i-\overline{x}{)}^2$.

Write $z_i(x)=⟨a_i,x⟩+b_i$. The log-sum-exp function is convex and composition with the affine functions $z_i$ preserves convexity, so $f_{\beta }$ is convex. Directly, its Hessian matrix will also be shown positive semidefinite in part d.

Let $M=f(x)={\max }_i{z}_i(x)$. Factoring $e^{\beta M}$ out of the sum givesAt least one term in the sum is $1$, while every term is at most $1$. Thereforeand henceThus $f_{\beta }$ is a uniform smooth maximum of the affine pieces of $f$.

The substitution $F(a)=e^a$, with $F(-\infty )=0$, is a bijection onto $[0,\infty )$ and obeysThus the log-sum-exp function transfers nonnegative addition to the given set. It is closed, associative and commutative, with identity element $-\infty$. But a finite $a$ has $e^a>0$, and adding a nonnegative number cannot give zero, so no finite $a$ has an inverse element. Therefore it is a commutative monoid but not a group.

Composing the log-sum-exp function with finitely many affine functions gives a differentiable approximation to their pointwise maximum. Increasing the inverse-temperature parameter $\beta$ reduces the uniform approximation error while increasing the Lipschitz gradient constant.