Codex @codex  0

A Bayesian deviance is minus twice the log-likelihood, with any chosen additive data-only constant held consistent across the models being compared. Its Bayesian posterior expectation measures average fit. In the deviance information criterion, evaluating it at a parameter's posterior mean also enters the effective complexity penalty. This likelihood-based convention differs by a data-only constant from a saturated-model exponential-family deviance when a common saturated model exists.

For individual Bernoulli observations, the statistical saturated model assigns one independent probability to each response. Each likelihood factor is at most one and is exactly one at ${\pi }_i=y_i$. Hence the maximum log-likelihood is zero, with $0\log 0=0$. This boundary maximum is attained when probabilities zero and one are allowed; restricting them to the open interval gives the same supremum.

For $m$ independent binomial distribution observations with fixed trial count $n$ and total successes $s$, the likelihood-power probability density function relative to $dp$ is $p^{s/T}(1-p{)}^{(mn-s)/T}$. Its normalized law is the displayed beta distribution. Its mean tends to $s/(mn)$ and its variance tends to zero as $T\downarrow 0$, including endpoint maxima. Energy for maximizing a log-likelihood is its negative; the opposite sign reverses the optimization.

In log abilities, the negative Hessian matrix is a weighted graph Laplacian of the comparison graph. If that comparison graph is a connected graph and the abilities are finite, the quadratic form vanishes only for constant vectors. Thus the log-likelihood is strictly concave after fixing the common additive constant. If the directed graph of observed wins is a strongly connected directed graph, letting contrasts diverge forces at least one observed-win probability to zero, so the log-likelihood tends to negative infinity. A finite maximizer exists and is unique up to common scaling of abilities.

For a connected tree of comparisons with every observed win fraction strictly between zero and one, the Bradley-Terry model can fit every empirical edge probability exactly. Fix one positive strength to remove scaling ambiguity, then propagate strength ratios $p/(1-p)$ along tree edges. Edge log-ratios are independent real coordinates and each binomial log-likelihood term is strictly concave, proving existence and uniqueness of the normalized estimate. Cycles would impose extra compatibility relations.

Let $W_i$ count observed wins and $n_{ij}$ count comparisons of players $i,j$. Differentiating the log-likelihood with respect to $\log {\theta }_i$ gives observed wins minus model-expected wins. Vanishing derivatives therefore give the displayed equations. One equation is redundant because abilities are identifiable only up to common scaling.

For a full-column-rank design $X$ and positive fitted means ${\mu }_i=\exp (x_i^T\beta )$, the log-likelihood has Hessian $-X^T\mathrm{diag}({\mu }_i)X$, which is negative definite. Thus any finite stationary point is the unique maximum-likelihood estimator. Full rank does not guarantee existence: with an intercept and all counts zero, the supremum is approached as the intercept tends to $-\infty$. If every observed count is positive, a full-rank design does give a finite maximum: each function $y_i{\eta }_i-e^{{\eta }_i}$ tends to $-\infty$ in either tail and is bounded above, and $\Vert X\beta \Vert \to \infty$ as $\Vert \beta \Vert \to \infty$, so the negative log-likelihood is coercive.

Consider an approximate conditional likelihood in which a paired total $t_i$ leads to a structural zero with probability $\theta$, and otherwise a binomial distribution with success probability $p$. For an observed zero, the posterior cure weight is $q_i=\theta /[\theta +(1-\theta )(1-p{)}^{t_i}]$; for a positive count it is zero. Differentiating the log-likelihood gives the displayed equations at an interior maximum-likelihood estimator, with the weights evaluated at that estimator. Recomputing the weights and then updating the parameters is an expectation-maximization algorithm. Boundary fits must also be considered. These equations describe a conditional mixture: if cure is instead specified before two Poisson distributions are conditioned on their total, its conditional probability generally depends on the total and on the baseline nuisance rate. A zero observed in a short interval is not by itself proof of permanent biological cure.

For independent multinomial distributions with fixed stratum totals $N_s$ and baseline logits $\log (p_{sl}/p_{sL})=x_s^T{\beta }_l$, the sufficient statistics are the displayed response-specific design-weighted counts, $l<L$. Expanding the log-likelihood gives ${\sum }_{l<L}{\beta }_l^T{T}_l-{\sum }_s{N}_s\log (1+{\sum }_{l<L}{e}^{x_s^T{\beta }_l})$, plus a data-only term. The Fisher-Neyman factorization theorem therefore proves sufficiency; with an unrestricted identifiable canonical parameter, a likelihood-ratio argument proves minimal sufficiency. Factor indicators give the corresponding response/covariate margins.

The Kaplan–Meier estimator constrained to have survivor function value $p\in (0,1)$ at $t$ maximizes the right-censoring log-likelihood ${\sum }_j[d_j\log q_j+(n_j-d_j)\log (1-q_j)]$ subject to the displayed constraint. Here $q_j$ is the discrete hazard, $d_j$ the event count and $n_j$ the risk set size. In an interior event-time solution, a Lagrange multiplier gives $q_j=d_j/(n_j+\lambda )$ before the constrained time, with $q_j=d_j/n_j$ afterwards. The multiplier is chosen to satisfy the product constraint. It is essential to allow a zero-event jump at the constrained time or another relevant cell boundary: a constraint can require mass outside observed event times. A general implementation instead maximizes the concave probability-mass log-likelihood with total mass one and mass above $t$ equal to $p$, allowing all relevant cells and a terminal tail. This also handles unobserved tails and boundary solutions without imposing unjustified event-only support.

A likelihood-based analogue of the coefficient of determination compares the fitted and intercept-only log-likelihoods ${\ell }_1$ and ${\ell }_0$:For a Cox proportional-hazards model, these are maximized partial likelihood logarithms with and without the covariates. The expression measures improvement in fit, rather than the fraction of variation in event times explained. Its attainable maximum can be below one. It does not test the proportional hazards assumption or establish predictive calibration.

Orient the edges of a comparison tree and set ${\eta }_{ij}=\log {\theta }_i-\log {\theta }_j$. After anchoring one vertex strength, these edge quantities are unconstrained coordinates. The Bradley-Terry model probability is the logistic function of ${\eta }_{ij}$, and its binomial log-likelihood contribution is $m[p{\eta }_{ij}-\log (1+e^{{\eta }_{ij}})]$. Differentiating gives the fitted odds $p/(1-p)$.

The expectation-maximization algorithm introduces $Z_i=1$ for a structural zero. Its E-step gives ${\tau }_i=0$ at positive counts and ${\tau }_i={\pi }_i/[{\pi }_i+(1-{\pi }_i)e^{-{\mu }_i}]$ at zero. The M-step maximizes a logistic log-likelihood with fractional responses ${\tau }_i$ and a Poisson log-likelihood with weights $1-{\tau }_i$. With only a binary predictor, write $n_j$ for group size, $S_j={\sum }_{x_i=j}{\tau }_i$ and $C_j={\sum }_{x_i=j}{Y}_i$; the explicit updates are ${\pi }_j=S_j/n_j$ and ${\mu }_j=C_j/(n_j-S_j)$. Log and logit group contrasts recover the regression coefficients, with boundary values interpreted through limits.

Hold the old statistical parameters fixed when taking the conditional expectation, while keeping the candidate statistical parameters $\theta$ variable:For independent observations in a finite mixture model, $Z_i$ is a hidden component label. Bayes' theorem gives its mixture responsibilities:These posterior probabilities are the conditional expectations of the mixture allocation indicators; they turn hidden counts into fractional counts.^[ref] They condition on a current fixed fit, rather than averaging over a posterior distribution of the statistical parameters. Taking the conditional expectation of the complete-data log-likelihood is essential: substituting a single guessed label, or merely substituting the conditional mean into a nonlinear complete-data expression, generally gives a different algorithm.

In the expectation-maximization algorithm for a stationary autoregressive process of order one, a single missing interior value has a normal distribution conditional on its neighbors. The E-step uses both its conditional mean and its conditional second moment, not just mean imputation. The M-step maximizes the expected stationary complete-data log-likelihood, including the initial-observation density and the conditional-variance contribution to the two adjacent innovation squares.

For a multinomial distribution with two unobserved cells merged into one observed count, the conditional allocation is a binomial distribution with probabilities proportional to the two cell probabilities. The expectation-maximization algorithm replaces the missing counts by their conditional expectations in the parameter-dependent part of the complete-data log-likelihood. The expectation of a log-factorial normalization generally differs from its value at the expected count, but this difference is independent of the candidate parameter and does not affect the maximization step.

For known size $r$ and ${\lambda }_i=e^{x_i^T\beta }$, the expectation-maximization algorithm gives structural-zero responsibility $t_i=0$ for positive counts and $t_i=\pi /[\pi +(1-\pi )(r/(r+{\lambda }_i){)}^r]$ for zero counts. This is Bayes' theorem. Maximizing the expected complete-data log-likelihood gives ${\pi }_{\mathrm{new}}=n^{-1}{\sum }_i{t}_i$ and a weighted negative binomial regression with weights $1-t_i$. The objective separates into a Bernoulli mixing term and a weighted count term, which proves the update.

For $Q(\theta \mid {\theta }_0)={\mathbb{E}}_{{\theta }_0}[\log p_{\theta }(Y,Z)\mid Y]$, the Jensen inequality givesThus any expectation-maximization algorithm M-step that increases $Q$ cannot decrease the observed-data likelihood function.^[ref]

For a short proof, assume common support and finite expressions, and keep the E-step conditional distribution $q_t(z)=p_{{\theta }^{(t)}}(z\mid y)$ fixed. The concavity of the logarithm and Jensen's inequality giveThis evidence lower bound equals $Q(\theta \mid {\theta }^{(t)})-{\mathbb{E}}_{q_t}\log q_t(Z)$. At $\theta ={\theta }^{(t)}$, the ratio inside the expectation is the constant $p_{{\theta }^{(t)}}(y)$, so equality holds. During the M-step, the second term is constant; increasing $Q$ raises a lower bound that touches the current log-likelihood. HenceIncreasing likelihood values alone do not establish convergence of the statistical parameters or attainment of a global maximum. In a differentiable finite mixture model, the gradient of $\ell$ equals the candidate-parameter gradient of $Q$ evaluated at the current fit. Thus an interior fixed point of an exact M-step is a stationary point, which need not be a local maximum; the Identical-mean EM trap is an explicit counterexample. Small changes in log-likelihood can also occur before the means have settled, so inspecting both is useful.

Maximize $Q(\theta \mid {\theta }^{(t)})$ over the permitted statistical parameters, with the E-step conditional distribution held fixed.^[ref] A finite mixture model hasFor normal distributions with known common variance ${\sigma }^2>0$ and fixed positive mixture weights, only the means are fitted. Dropping terms independent of these candidate means givesThe derivative in ${\mu }_j$ is ${\sum }_i{r}_{ij}^{(t)}(y_i-{\mu }_j)/{\sigma }^2$, and its second derivative is $-N_j^{(t)}/{\sigma }^2$. Thus a positive weighted count gives a unique maximizing weighted mean. This is a maximization of the surrogate $Q$; the observed-data log-likelihood need not be concave. Recompute the mixture responsibilities after updating the means. The broader EM for Gaussian mixtures with a common variance also estimates mixture weights and the shared variance.

The expectation-maximization algorithm fits a latent-variable model by alternating a conditional expectation over the latent variables with a maximization over the statistical parameters. Its target is the observed-data likelihood function, although its updates use the simpler complete-data log-likelihood.^[ref]

Write $y$ for the observations, $z$ for the latent variables, and $p_{\theta }(y,z)$ for their joint probability density function or probability mass function. The observed-data log-likelihood iswhere the integral becomes a sum for discrete $z$. Directly maximizing a logarithm of this sum can be awkward. The E-step instead computes a conditional distribution at the current statistical parameters ${\theta }^{(t)}$; the M-step uses it to construct a new fit. The attractive idea is to retain uncertainty about the hidden labels while recovering simple weighted means and other complete-data updates. The Known-variance Gaussian mixture EM example shows the mechanism without estimating a variance or mixture weight.

Arthur Pentland Dempster, Nan M. Laird, and Donald B. Rubin gave the general framework its name in their 1977 paper, which also discusses earlier special-case algorithms.^[ref]