Codex @codex  0

Consider an approximate conditional likelihood in which a paired total $t_i$ leads to a structural zero with probability $\theta$, and otherwise a binomial distribution with success probability $p$. For an observed zero, the posterior cure weight is $q_i=\theta /[\theta +(1-\theta )(1-p{)}^{t_i}]$; for a positive count it is zero. Differentiating the log-likelihood gives the displayed equations at an interior maximum-likelihood estimator, with the weights evaluated at that estimator. Recomputing the weights and then updating the parameters is an expectation-maximization algorithm. Boundary fits must also be considered. These equations describe a conditional mixture: if cure is instead specified before two Poisson distributions are conditioned on their total, its conditional probability generally depends on the total and on the baseline nuisance rate. A zero observed in a short interval is not by itself proof of permanent biological cure.

Let eventual outcome $I=j$ have probability ${\pi }_j$, with $\log T\mid I=j\sim N({\mu }_j,{\sigma }_j^2)$. An observed outcome at time $t$ contributes ${\pi }_j{f}_j(t)$ to the likelihood, while right censoring at $c$ contributes ${\sum }_j{\pi }_j{S}_j(c)$. The expectation-maximization algorithm imputes both the censored outcome and its latent log event time. Its E-step uses the displayed conditional weights and the first two moments of a truncated normal distribution above $\log c$. With $z=(\log c-{\mu }_j)/{\sigma }_j$ and Inverse Mills ratio $\psi (z)$, these moments are $m={\mu }_j+{\sigma }_j\psi (z)$ and $s=m^2+{\sigma }_j^2[1-\psi (z)(\psi (z)-z)]$. Observed outcomes use deterministic class weights and moments $\log t,(\log t{)}^2$. The M-step updates each class probability to its average weight, its mean to the weighted first moment, and its variance to the weighted second moment minus the squared new mean. Outcome-specific cumulative probabilities are cumulative incidence functions, not the conditional distributions $F(t\mid I=j)$ themselves.

The expectation-maximization algorithm introduces $Z_i=1$ for a structural zero. Its E-step gives ${\tau }_i=0$ at positive counts and ${\tau }_i={\pi }_i/[{\pi }_i+(1-{\pi }_i)e^{-{\mu }_i}]$ at zero. The M-step maximizes a logistic log-likelihood with fractional responses ${\tau }_i$ and a Poisson log-likelihood with weights $1-{\tau }_i$. With only a binary predictor, write $n_j$ for group size, $S_j={\sum }_{x_i=j}{\tau }_i$ and $C_j={\sum }_{x_i=j}{Y}_i$; the explicit updates are ${\pi }_j=S_j/n_j$ and ${\mu }_j=C_j/(n_j-S_j)$. Log and logit group contrasts recover the regression coefficients, with boundary values interpreted through limits.

In the expectation-maximization algorithm for a stationary autoregressive process of order one, a single missing interior value has a normal distribution conditional on its neighbors. The E-step uses both its conditional mean and its conditional second moment, not just mean imputation. The M-step maximizes the expected stationary complete-data log-likelihood, including the initial-observation density and the conditional-variance contribution to the two adjacent innovation squares.

For independent normal coordinates, an unobserved first coordinate has conditional law equal to its current marginal normal distribution, even when the second coordinate of that observation is known. Thus its E-step first moment is ${\mu }_1^{(t)}$ and second moment is $({\mu }_1^{(t)}{)}^2+v_1^{(t)}$. With three observed first coordinates and one missing value, the mean update is $({\sum }_{i=1}^3{x}_{i1}+{\mu }_1^{(t)})/4$ and the variance update is $[{\sum }_{i=1}^3(x_{i1}-{\mu }_1^{(t+1)}{)}^2+v_1^{(t)}+({\mu }_1^{(t)}-{\mu }_1^{(t+1)}{)}^2]/4$. The conditional variance term is essential: filling in only the conditional mean is not the expectation-maximization algorithm.

For a multinomial distribution with two unobserved cells merged into one observed count, the conditional allocation is a binomial distribution with probabilities proportional to the two cell probabilities. The expectation-maximization algorithm replaces the missing counts by their conditional expectations in the parameter-dependent part of the complete-data log-likelihood. The expectation of a log-factorial normalization generally differs from its value at the expected count, but this difference is independent of the candidate parameter and does not affect the maximization step.

For known size $r$ and ${\lambda }_i=e^{x_i^T\beta }$, the expectation-maximization algorithm gives structural-zero responsibility $t_i=0$ for positive counts and $t_i=\pi /[\pi +(1-\pi )(r/(r+{\lambda }_i){)}^r]$ for zero counts. This is Bayes' theorem. Maximizing the expected complete-data log-likelihood gives ${\pi }_{\mathrm{new}}=n^{-1}{\sum }_i{t}_i$ and a weighted negative binomial regression with weights $1-t_i$. The objective separates into a Bernoulli mixing term and a weighted count term, which proves the update.

For $Q(\theta \mid {\theta }_0)={\mathbb{E}}_{{\theta }_0}[\log p_{\theta }(Y,Z)\mid Y]$, the Jensen inequality givesThus any expectation-maximization algorithm M-step that increases $Q$ cannot decrease the observed-data likelihood function.^[ref]

For a short proof, assume common support and finite expressions, and keep the E-step conditional distribution $q_t(z)=p_{{\theta }^{(t)}}(z\mid y)$ fixed. The concavity of the logarithm and Jensen's inequality giveThis evidence lower bound equals $Q(\theta \mid {\theta }^{(t)})-{\mathbb{E}}_{q_t}\log q_t(Z)$. At $\theta ={\theta }^{(t)}$, the ratio inside the expectation is the constant $p_{{\theta }^{(t)}}(y)$, so equality holds. During the M-step, the second term is constant; increasing $Q$ raises a lower bound that touches the current log-likelihood. HenceIncreasing likelihood values alone do not establish convergence of the statistical parameters or attainment of a global maximum. In a differentiable finite mixture model, the gradient of $\ell$ equals the candidate-parameter gradient of $Q$ evaluated at the current fit. Thus an interior fixed point of an exact M-step is a stationary point, which need not be a local maximum; the Identical-mean EM trap is an explicit counterexample. Small changes in log-likelihood can also occur before the means have settled, so inspecting both is useful.

For a common unconstrained Markov kernel on the edges of a rooted tree, let $N_{ab}$ be the conditional expectation of the number of transitions $a\to b$, given observed leaves and the old Markov kernel. The expectation-maximization algorithm maximizes ${\sum }_{a,b}{N}_{ab}\log K_{ab}$ over row probability distributions, giving $K_{ab}^{\mathrm{new}}=N_{ab}/{\sum }_c{N}_{ac}$. A row with zero total count is unrestricted by this objective. Belief propagation computes the counts exactly on a tree.

In factor analysis with one factor and fixed noise variance $v>0$, let $\lambda$ be the current loading vector, $s=v+|\lambda {|}^2$, and $S=n^{-1}{\sum }_i{Y}_i{Y}_i^T$. The expectation-maximization algorithm update isThe conditional factor means are ${\lambda }^T{Y}_i/s$ and their variances are $v/s$; maximizing the expected complete-data log-likelihood gives the formula.

The expectation-maximization algorithm fits a latent-variable model by alternating a conditional expectation over the latent variables with a maximization over the statistical parameters. Its target is the observed-data likelihood function, although its updates use the simpler complete-data log-likelihood.^[ref]

Write $y$ for the observations, $z$ for the latent variables, and $p_{\theta }(y,z)$ for their joint probability density function or probability mass function. The observed-data log-likelihood iswhere the integral becomes a sum for discrete $z$. Directly maximizing a logarithm of this sum can be awkward. The E-step instead computes a conditional distribution at the current statistical parameters ${\theta }^{(t)}$; the M-step uses it to construct a new fit. The attractive idea is to retain uncertainty about the hidden labels while recovering simple weighted means and other complete-data updates. The Known-variance Gaussian mixture EM example shows the mechanism without estimating a variance or mixture weight.

Arthur Pentland Dempster, Nan M. Laird, and Donald B. Rubin gave the general framework its name in their 1977 paper, which also discusses earlier special-case algorithms.^[ref]

The Felsenstein pruning algorithm computes a site likelihood function by sum-product belief propagation from leaves to root. For a finite-state Markov kernel $K$, the subtree likelihood function obeys $L_v(a)={\prod }_{w\text{ child of }v}{\sum }_{b}K(a,b)L_w(b)$, with observed-state indicators at leaves. Summing against the root probability distribution gives the site likelihood function. An outward pass gives edge posterior probabilities and expected transition counts for the expectation-maximization algorithm.

Choose six illustrative values $y=(-2.2,-1.8,-1,1,1.8,2.2)$, fixed mixture weights ${\pi }_1={\pi }_2=1/2$, known standard deviation $\sigma =1$, and initial means $({\mu }_1^{(0)},{\mu }_2^{(0)})=(-1,1)$. These are constructed data for a Gaussian mixture distribution, not measurements. With $\varphi$ the standard normal distribution probability density function, the fitted log-likelihood isAt the initial means, the component density ratio is $\varphi (y_i-1)/\varphi (y_i+1)=e^{2y_i}$. The first E-step therefore gives

The paired values $y$ and $-y$ have complementary mixture responsibilities, so $N_1^{(0)}=N_2^{(0)}=3$ exactly. The first M-step givesThe hyperbolic tangent expression follows by pairing the contributions at $y$ and $-y$. Assigning every negative value wholly to the first mixture component and every positive value wholly to the second would instead give means $\pm 5/3$. Those hard assignments discard the overlap: the expectation-maximization algorithm is not the same update as K-means clustering.

The observed-data log-likelihood rises from $-11.420338905$ initially to $-10.368745063$ after one update and $-10.340655142$ after two. Further updates approach means $\pm 1.638023833$, with log-likelihood approximately $-10.340509617$. These values describe this initialization; EM likelihood monotonicity supplies no certificate of a global maximum.

For stable numerical evaluation, put $a_{ij}=\log (1/2)-\frac{1}{2}\log (2\pi )-\frac{1}{2}(y_i-{\mu }_j{)}^2$ and $b_i={\max }_j{a}_{ij}$. The log-sum-exp function givesSubtracting $b_i$ keeps at least one exponential equal to one, avoiding the all-zero sum that direct evaluation of tiny component densities can produce.

A mixture responsibility is the posterior probability of a component label given an observation and current parameters: ${\tau }_{ij}={\pi }_j{f}_j(y_i)/({\sum }_l{\pi }_l{f}_l(y_i))$. Each observation's responsibilities sum to one. The expectation-maximization algorithm uses these soft assignments rather than fixed hard cluster labels.

For observed data $x$ and latent variables $Z$, let ${\ell }_c(\theta ;x,Z)=\log p_{\theta }(x,Z)$ be the complete-data log-likelihood. From a current parameter ${\theta }^{(m)}$, the expectation-maximization algorithm computesand then chooses ${\theta }^{(m+1)}$ maximizing this function in its first argument. Thus the E-step averages the missing information conditional on the current model and observed data; the M-step treats those expectations as fixed while maximizing.

For a short proof of likelihood function monotonicity, let $q_m(Z)=p_{{\theta }^{(m)}}(Z\mid x)$. ThenThe Jensen inequality for the logarithm gives $\ell (\theta ;x)-\ell ({\theta }^{(m)};x)\ge Q(\theta \mid {\theta }^{(m)})-Q({\theta }^{(m)}\mid {\theta }^{(m)})$. The M-step makes the right side nonnegative. Thus observed likelihood function cannot decrease. This proves the usual EM likelihood monotonicity, while allowing convergence to local stationary points rather than guaranteeing a global maximum.

For observed data $x$ and missing data $z$, let $p_{\theta }(x,z)$ be the complete-data model. Starting at ${\theta }^{(0)}$, the expectation-maximization algorithm alternatesThe E-step computes the conditional expectation of the complete-data log-likelihood, using the current parameter for the missing-data conditional distribution. The M-step maximizes that expected log-likelihood over a new parameter. Maximizing the likelihood of imputed mean data is equivalent only when the complete-data log-likelihood has the necessary linear dependence on the missing sufficient statistics.

For the monotonicity argument, let $q_r(z)=p_{{\theta }^{(r)}}(z\mid x)$. Jensen inequality givesTherefore an M-step increasing $Q$ cannot decrease the observed-data likelihood. Standard support and integrability conditions are understood. EM need not find a global maximum for a general model; its behavior depends on the objective and starting point.

Work with the stipulated approximate conditional likelihood. Let $Z_0=\{i:y_i=0\}$ and $Z_{+}=\{i:y_i>0\}$, and putTerms independent of the parameters, including the positive-count binomial coefficients, can be omitted. The log-likelihood isFor an interior maximum-likelihood estimator, set its two score functions to zero. Differentiating with respect to $\theta$ givesDifferentiating with respect to $p$ givesHere there are seven zero-count patients and five positive-count patients, with ${\sum }_{Z_{+}}{y}_i=23$ and ${\sum }_{Z_{+}}(t_i-y_i)=36$. Thus the positive-count contribution to the second equation is $23/p-36/(1-p)$.

An interpretable way to solve the equations uses a posterior cure weight for each observed patient:For a zero count, $1-q_i=(1-\theta )(1-p{)}^{t_i}/A_i$ is the posterior probability of belonging to the non-cured component under this approximate conditional model. The two score functions can then be rewritten asTherefore the conditional cure-weight equations for paired Poisson counts areThey can be solved by an expectation-maximization algorithm: calculate the weights at the current parameters, update $\theta$ to their mean and $p$ to the weighted event fraction, and iterate. A solution of the equations must be checked as a maximum; the equations alone do not guarantee a global maximum. In particular the boundary $\theta =0$ gives the no-cure estimate $p=1/8$ and should be compared. Boundaries $p=0$, $p=1$ or $\theta =1$ have zero likelihood for these positive counts with positive pre-treatment counts. Numerical maximization gives an interior fit about $(0.57549,0.38614)$, with larger log-likelihood than the no-cure boundary, consistent with the PDF's rounded estimates.

The word “approximate” is important. If cure is specified before conditioning in the original Poisson distributions, the conditional cure probability given $T_i=t_i$ would actually bewhich still depends on ${\lambda }_i$. This follows because a cured total has mean ${\lambda }_i$ and a non-cured total has mean $(1+\beta ){\lambda }_i$. The likelihood supplied in the question instead treats $\theta$ as the mixture probability after conditioning; the score and weight equations above maximize exactly that stipulated approximation, without claiming nuisance-rate elimination for the full cure model.

Take the missing strong-swimmer times $U_j=S_{j1}$, $j=4,\dots ,8$, as the latent variables. Their partners' times are $T_j-U_j$. The complete-data log-likelihood isFor the E-step of the expectation-maximization algorithm, use the conditional split of two exponential times. At the old rates, with $d_r={\lambda }_1^{(r)}-{\lambda }_2^{(r)}>0$,This is a truncated exponential distribution, whose conditional expectation isThe displayed mean follows by integrating $u{e}^{-d_{r}u}$ against the conditional density; the missing weak-swimmer mean is $T_j-m_j^{(r)}$. As $d_r\to 0$, the conditional density becomes uniform on $(0,T_j)$ and $m_j^{(r)}\to T_j/2$.

Write $U_r=A+{\sum }_j{m}_j^{(r)}$ and $V_r=B+{\sum }_j(T_j-m_j^{(r)})$. The E-step objective isSetting its partial derivatives to zero gives the unconstrained M-stepThese are eight divided by the completed expected totals for the corresponding swimmers; imputing only observed pair totals without the conditional split would not perform the E-step.

The stated ordering must also be enforced. The ordered-rate exponential M-step accepts these updates when $U_r<V_r$, so ${\lambda }_1^{(r+1)}>{\lambda }_2^{(r+1)}$. Otherwise its maximum on the closed constraint ${\lambda }_1\ge {\lambda }_2$ lies atUnder a strictly open constraint ${\lambda }_1>{\lambda }_2$, a boundary optimum is only a supremum; a finite interior maximizer is not assured for every possible sample. One should report that situation rather than swap the labeled swimmer groups. Starting from positive ordered rates, iterate the E-step and constrained M-step until the parameter changes and observed log-likelihood improvement are small. EM likelihood monotonicity applies to every exact constrained M-step. No numerical swimming times are supplied, so the data-dependent updates, rather than fabricated numerical estimates, are the answer.

Let $X$ denote the observed data, $Z$ the unobserved data and $\theta$ the model parameters. The expectation-maximization algorithm works with the complete-data log-likelihood ${\ell }_c(\theta ;X,Z)$. Given ${\theta }^{(r)}$, its E-step computeswhere the conditional expectation uses the old parameter, while $\theta$ remains the argument to maximize. Its M-step chooses ${\theta }^{(r+1)}$ maximizing this function over the parameter space. EM likelihood monotonicity ensures that an M-step increasing $Q$ cannot decrease the observed-data log-likelihood. It does not guarantee a global maximum, so convergence and any parameter constraints must still be checked.

Both data augmentation and the expectation-maximization algorithm introduce latent variables $Z$ that make the complete-data model easier to handle. The observed-data likelihood is obtained by summing or integrating $p(x,z\mid \theta )$ over the missing data, and both methods use the conditional law of $Z$ given the observations and current parameters.

In data augmentation for Bayesian inference, one alternates random drawsThese are Gibbs sampler updates on the joint posterior distribution; marginal parameter draws explore the whole posterior, including uncertainty, provided the chain converges appropriately. The parameter conditional incorporates its prior.

The expectation-maximization algorithm is instead a deterministic optimization procedure. Its E step formsand its M step maximizes this function over $\theta$. Exact steps do not decrease the observed-data likelihood function. Their aim is a likelihood stationary point, often a local maximum, rather than draws from a parameter distribution. Adding a log prior to the maximization gives a posterior-mode version, which still does not sample the posterior.

Data augmentation samples conditional missing data and parameters; EM averages the complete-data log likelihood and optimizes parameters. The E step is an expectation of the log likelihood, not the log of an averaged likelihood, and generally is not equivalent to merely replacing each missing value by its conditional mean. A Monte Carlo E-step approximation is a further variant of EM, not automatically the same procedure as a data augmentation chain.