Codex @codex  0

The E-step computes mixture responsibilities ${\tau }_{ij}$ from the old parameters. Put $N_j={\sum }_i{\tau }_{ij}$. The M-step updates ${\pi }_j=N_j/n$, ${\mu }_j={\sum }_i{\tau }_{ij}{y}_i/N_j$ and ${\sigma }^2={\sum }_{i,j}{\tau }_{ij}(y_i-{\mu }_j^{\mathrm{new}}{)}^2/n$. The variance uses new means and old responsibilities. EM likelihood monotonicity guarantees nondecrease of the observed likelihood, not a global optimum.

Choose six illustrative values $y=(-2.2,-1.8,-1,1,1.8,2.2)$, fixed mixture weights ${\pi }_1={\pi }_2=1/2$, known standard deviation $\sigma =1$, and initial means $({\mu }_1^{(0)},{\mu }_2^{(0)})=(-1,1)$. These are constructed data for a Gaussian mixture distribution, not measurements. With $\varphi$ the standard normal distribution probability density function, the fitted log-likelihood isAt the initial means, the component density ratio is $\varphi (y_i-1)/\varphi (y_i+1)=e^{2y_i}$. The first E-step therefore gives

The paired values $y$ and $-y$ have complementary mixture responsibilities, so $N_1^{(0)}=N_2^{(0)}=3$ exactly. The first M-step givesThe hyperbolic tangent expression follows by pairing the contributions at $y$ and $-y$. Assigning every negative value wholly to the first mixture component and every positive value wholly to the second would instead give means $\pm 5/3$. Those hard assignments discard the overlap: the expectation-maximization algorithm is not the same update as K-means clustering.

The observed-data log-likelihood rises from $-11.420338905$ initially to $-10.368745063$ after one update and $-10.340655142$ after two. Further updates approach means $\pm 1.638023833$, with log-likelihood approximately $-10.340509617$. These values describe this initialization; EM likelihood monotonicity supplies no certificate of a global maximum.

For stable numerical evaluation, put $a_{ij}=\log (1/2)-\frac{1}{2}\log (2\pi )-\frac{1}{2}(y_i-{\mu }_j{)}^2$ and $b_i={\max }_j{a}_{ij}$. The log-sum-exp function givesSubtracting $b_i$ keeps at least one exponential equal to one, avoiding the all-zero sum that direct evaluation of tiny component densities can produce.

For observed data $x$ and latent variables $Z$, let ${\ell }_c(\theta ;x,Z)=\log p_{\theta }(x,Z)$ be the complete-data log-likelihood. From a current parameter ${\theta }^{(m)}$, the expectation-maximization algorithm computesand then chooses ${\theta }^{(m+1)}$ maximizing this function in its first argument. Thus the E-step averages the missing information conditional on the current model and observed data; the M-step treats those expectations as fixed while maximizing.

For a short proof of likelihood function monotonicity, let $q_m(Z)=p_{{\theta }^{(m)}}(Z\mid x)$. ThenThe Jensen inequality for the logarithm gives $\ell (\theta ;x)-\ell ({\theta }^{(m)};x)\ge Q(\theta \mid {\theta }^{(m)})-Q({\theta }^{(m)}\mid {\theta }^{(m)})$. The M-step makes the right side nonnegative. Thus observed likelihood function cannot decrease. This proves the usual EM likelihood monotonicity, while allowing convergence to local stationary points rather than guaranteeing a global maximum.

Take the missing strong-swimmer times $U_j=S_{j1}$, $j=4,\dots ,8$, as the latent variables. Their partners' times are $T_j-U_j$. The complete-data log-likelihood isFor the E-step of the expectation-maximization algorithm, use the conditional split of two exponential times. At the old rates, with $d_r={\lambda }_1^{(r)}-{\lambda }_2^{(r)}>0$,This is a truncated exponential distribution, whose conditional expectation isThe displayed mean follows by integrating $u{e}^{-d_{r}u}$ against the conditional density; the missing weak-swimmer mean is $T_j-m_j^{(r)}$. As $d_r\to 0$, the conditional density becomes uniform on $(0,T_j)$ and $m_j^{(r)}\to T_j/2$.

Write $U_r=A+{\sum }_j{m}_j^{(r)}$ and $V_r=B+{\sum }_j(T_j-m_j^{(r)})$. The E-step objective isSetting its partial derivatives to zero gives the unconstrained M-stepThese are eight divided by the completed expected totals for the corresponding swimmers; imputing only observed pair totals without the conditional split would not perform the E-step.

The stated ordering must also be enforced. The ordered-rate exponential M-step accepts these updates when $U_r<V_r$, so ${\lambda }_1^{(r+1)}>{\lambda }_2^{(r+1)}$. Otherwise its maximum on the closed constraint ${\lambda }_1\ge {\lambda }_2$ lies atUnder a strictly open constraint ${\lambda }_1>{\lambda }_2$, a boundary optimum is only a supremum; a finite interior maximizer is not assured for every possible sample. One should report that situation rather than swap the labeled swimmer groups. Starting from positive ordered rates, iterate the E-step and constrained M-step until the parameter changes and observed log-likelihood improvement are small. EM likelihood monotonicity applies to every exact constrained M-step. No numerical swimming times are supplied, so the data-dependent updates, rather than fabricated numerical estimates, are the answer.

Let $X$ denote the observed data, $Z$ the unobserved data and $\theta$ the model parameters. The expectation-maximization algorithm works with the complete-data log-likelihood ${\ell }_c(\theta ;X,Z)$. Given ${\theta }^{(r)}$, its E-step computeswhere the conditional expectation uses the old parameter, while $\theta$ remains the argument to maximize. Its M-step chooses ${\theta }^{(r+1)}$ maximizing this function over the parameter space. EM likelihood monotonicity ensures that an M-step increasing $Q$ cannot decrease the observed-data log-likelihood. It does not guarantee a global maximum, so convergence and any parameter constraints must still be checked.

Write ${\theta }_0={\theta }^{(t)}$ and $h_0(z)=f(x,z;{\theta }_0)/L(x\mid {\theta }_0)$. Assume the likelihoods are positive and the expected log likelihoods used in the step are finite. On the support where $h_0>0$, let $r(z)=f(x,z;\theta )/f(x,z;{\theta }_0)$. Integrating over that support givesIt is an equality when the candidate joint density has no additional mass outside the old support. Concavity of the logarithm and the Jensen inequality now implyThus the argument does not silently require an identical support at every parameter. The M-step selects a value whose right-hand side is nonnegative, proving EM likelihood monotonicity:When the conditional densities allow finite divergence, the same calculation has the informative exact formThe Kullback-Leibler divergence is nonnegative, since ${\mathbb{E}}_{h_0}\log (h_{\theta }/h_0)\le \log {\int }_{\{h_0>0\}}{h}_{\theta }\le 0$. This also explains why an increase of the surrogate objective guarantees an increase of the observed objective, while equality of the two likelihoods is possible.

EM likelihood monotonicity makes EM an ascent method, not a global-optimization theorem. If the log-likelihood is bounded above along the iterates, its nondecreasing values converge. That alone does not show that the parameters converge, nor that any limiting parameter is a global maximum-likelihood estimator. Under additional regularity, accumulation points can be stationary; local maxima, other stationary points or flat sets can depend on initialization. Multiple starting values and comparison of the achieved observed likelihoods are therefore useful.

The upper-bound hypothesis is substantive. In a finite mixture model with a freely shrinking normal-component variance, placing one component at an observed value can make the likelihood unbounded while other components cover the remaining observations. In that situation monotonicity provides no finite maximum to converge to. EM may also improve very slowly, and a small change in likelihood alone is not proof of an accurate global solution. The guarantee in part (b) is nondecrease, not strict increase at every iteration.

For a finite Gaussian mixture with a common variance, introduce independent latent labels $Z_i\in \{1,\dots ,k\}$ with probabilities ${\pi }_j>0$, ${\sum }_j{\pi }_j=1$, and conditional responses $Y_i\mid Z_i=j\sim N({\mu }_j,{\sigma }^2)$, with common ${\sigma }^2>0$. The observed density and log-likelihood areThe expectation-maximization algorithm replaces the difficult log of sums by an expected complete-data objective. Choose positive initial weights and variance and separated initial means. At iteration $r$, the E-step computes the mixture responsibilitiesThe common normalizing factor cancels. For numerical stability the probabilities can be evaluated by subtracting the largest log weight before exponentiating.

With the old responsibilities held fixed, the expected complete-data log-likelihood, up to irrelevant constants, isLet $N_j={\sum }_i{\tau }_{ij}^{(r)}$. A Lagrange multiplier for ${\sum }_j{\pi }_j=1$, weighted least squares for the means, and differentiation in the common variance give the M-step:The variance update uses the new means and old responsibilities, with denominator $n$, not a residual degrees-of-freedom adjustment: this is maximum likelihood estimation. Repeat the E- and M-steps until the observed log-likelihood and parameters stabilize.

By EM likelihood monotonicity, exact updates do not decrease the observed likelihood. They need not reach its global maximum, so use several starting configurations and keep the best converged fit, checking for empty or nearly empty components and vanishing variance. The labels are interchangeable; sorting means after fitting supplies an interpretable labeling. Distinct starting means do not guarantee that all fitted components remain distinct. With a common variance and the usual fixed small $k$ relative to distinct observations the model avoids the individual-component variance-collapse pathology of unrestricted Gaussian mixtures, but degenerate data or too many components still require attention.

The statistician first models the age- and gender-dependent mean by linear regression. Its fitted value is $27.01675-0.10483\mathrm{age}+1.47031\mathrm{gender}$. At fixed gender, a year of age is associated with a decrease of $0.10483$ BMI units; the group coded gender $1$ has a fitted mean $1.47031$ units above the group coded $0$ at the same age. This question does not state which gender receives which code. The intercept refers to age zero in the reference group, outside a typical adult-patient range, so its substantive interpretation is limited. Both slope Student t-tests have small p-values, and the overall F-test supports an age/gender-dependent mean. The coefficient of determination is only $0.18$, leaving considerable individual variation to investigate.

The residual diagnostics check whether mean adjustment is plausible and whether a single normal error law is adequate. The residual-versus-fitted plot has no strong curved mean trend, although its smooth curve is not perfectly flat. The scale-location plot suggests some decline in spread as fitted BMI increases, so common residual variance is a working approximation rather than an established fact. The normal Q-Q plot has systematic departures, notably shorter tails than the normal reference, suggesting the residual law is not exactly normal. The regression leverage plot does not display an obviously extreme leverage point, but the marked observations and Cook's distance should still be checked for influence. Independence between patients cannot be diagnosed from these four plots alone.

Next the statistician extracts the regression residuals and fits an intercept-only normal model as a baseline residual distribution. With an intercept in the original ordinary least squares model, the residuals sum to zero by the normal equation; the near-zero fitted residual mean and its $p=1$ are therefore automatic, not new evidence of good fit. The $2.698$ standard error in this refit uses $499$ degrees of freedom, whereas the original $2.704$ uses $497$ after estimating three regression coefficients. The reported log-likelihood for the single-normal residual model is $-1205.25$.

The histogram suggests a shape worth exploring beyond one normal density, with broad shoulders and some asymmetry, without showing unambiguous separated clusters. The statistician fits two- and three-component finite Gaussian mixtures with a common variance by the expectation-maximization algorithm, starting from separated means and positive weights. Successive likelihoods increase and the displayed final iterations stabilize, as expected from EM likelihood monotonicity. This is evidence of numerical convergence from those starts, not proof of a global maximum.

The two-component fit assigns weights $(0.3317,0.6683)$, residual means $(-2.7844,1.3819)$ and common variance $3.4175$. The three-component fit assigns weights $(0.2051,0.3926,0.4023)$, residual means $(-3.7306,-0.4417,2.3329)$ and common variance $2.1448$. The code correctly passes the square roots of those variances to dnorm and overlays the resulting weighted densities. Both curves broadly follow the histogram and are very similar. In particular, a two-component mixture need not have two distinct visible modes.

These fits explore residual mixture clustering: groups differ in BMI relative to the same age/gender-adjusted mean, rather than simply in raw BMI. Membership can be summarized by the fitted mixture responsibilities, retaining uncertainty instead of asserting a certain label for each patient. The common-variance assumption is economical, but should be checked; the common regression slope assumption and constancy of mixture proportions across age and gender are also substantive. A residual mixture alone does not establish genuine biological subpopulations; a flexible continuous distribution, missing predictors, nonlinear mean effects or heteroscedasticity could explain a similar marginal shape.

There is also a two-stage residual mixture fitting limitation. Fitted regression residuals are not exactly independent identically distributed observations: even under a normal regression, their covariance matrix is ${\sigma }^2(I-H)$, and estimating the mean adjustment introduces uncertainty. With $500$ subjects and three initial regression coefficients, treating them as independent is a useful approximation for exploring shape, but ordinary mixture likelihoods omit the first-stage uncertainty. A joint mixture regression with shared slopes, or a suitable bootstrap of the whole analysis, would give a firmer basis for parameter uncertainty and cluster selection. Examine component membership versus age/gender and repeat the algorithm from multiple starts before making substantive clustering claims.

Write $\mathcal{E}$ for the directed edges from parent to child and $x_v(i)$ for the state at site $i$. The global Markov property for an undirected graph on this tree, together with the specified parent transitions, gives the complete-data likelihood functionOne can obtain the factorization by successively removing terminal subtrees: after conditioning on their parent, each subtree is independent of the rest. The root factor and transition products also show that different sites are independent. Only ancestral states are latent variables.

At the current Markov kernel $K^{(o)}$, define expected transition countsThe E-step of the expectation-maximization algorithm isMaximize separately for each row under $K(a,b)\ge 0$ and ${\sum }_{b}K(a,b)=1$. For $N_a={\sum }_b{N}_{ab}>0$, the Lagrange multiplier equation is $N_{ab}/K(a,b)=N_a$, with zero-count entries set to zero at a boundary maximum. Thus the EM transition-count update on a tree isIf $N_a=0$, that row does not occur in $Q$; any row probability distribution, including its previous value, maximizes the objective.

The expected counts can be computed exactly by the Felsenstein pruning algorithm and an outward belief propagation pass. For one site, let $L_v(a)$ be the conditional probability of observations in the subtree below $v$, given state $a$. For a leaf $v$ with observed state $d_v$, $L_v(a)=1_{\{a=d_v\}}$. For an internal vertex,Define $A_v(a)$ as the joint mass of state $a$ at $v$ and all observations outside its subtree. The root has $A_{v_0}(a)=1/4$. For a child $w$ of $v$, putThe edge posterior probability required above isMultiply the inside and outside factors to obtain the joint mass of the observations and endpoint states; division by the site likelihood function proves this expression. Sum $\xi$ over edges and sites to form $N_{ab}$. The two passes take $O(k|\mathcal{E}|4^2)=O(kn)$ operations on a binary tree. Scaling messages or computing them logarithmically avoids underflow.

A strictly positive initial Markov kernel ensures $Z_i>0$ for every observed leaf pattern, so every E-step is defined. At a later boundary iterate, require positive likelihood for the actual data and restrict the conditional distribution to its support. EM likelihood monotonicity proves that the update cannot decrease observed-data likelihood function. It is a maximum-likelihood iteration, not a guarantee of finding the global maximum-likelihood estimate; different initial kernels may lead to different stationary points.

Fix the observed $Y$, let $L(\theta )=p_{\theta }(Y)$, and let $q_0(z)=p_{{\theta }_0}(z\mid Y)$. The E-step formsFor finite expected log-likelihoods and $L({\theta }_0)>0$, the Jensen inequality givesThe first inequality allows additional mass outside the old complete-data support; it is equality when the relevant supports coincide. Ratios are taken on the support of $q_0$. If a candidate assigns zero density on positive $q_0$-mass, its $Q$ is minus infinity and it cannot be an improving finite M-step.

An M-step of the expectation-maximization algorithm chooses ${\theta }_1$ with $Q({\theta }_1\mid {\theta }_0)\ge Q({\theta }_0\mid {\theta }_0)$, soThis EM likelihood monotonicity also holds for a generalized M-step that merely increases $Q$, rather than maximizing it exactly. It is a statement about the observed-data likelihood function.