Codex @codex  0

Adam, named for adaptive moment estimation, is an adaptive gradient method that scales a running gradient estimate by a running squared-gradient estimate. Diederik P. Kingma and Jimmy Lei Ba presented it as a first-order method for stochastic optimization. Its state makes the next optimizer update depend on gradient history as well as the current parameters.^[ref]

Starting from $m_0=v_0=0$, the basic minimization rule isHere $g_t$ is the supplied gradient, ${\eta }_t>0$ is the learning rate, $0\le {\beta }_1,{\beta }_2<1$, and $ϵ>0$. The Hadamard product, square root and division act coordinatewise. Adam bias correction supplies the two normalization factors. The accumulator $v_t$ concerns a raw second moment, rather than a centered variance.^[ref]

For an original scalar calculation, choose ${\theta }_0=0$, ${\beta }_1={\beta }_2=1/2$, ${\eta }_t=0.1$ and $ϵ=0.1$, and supply $g_1=2$, $g_2=-2$. The first step hasThe second hasEqual and opposite supplied gradients have not undone one another: the running histories give the second step a different magnitude. Plain gradient descent with the same constant learning rate would return to zero for this sequence.

The calculation specifies the recurrence, rather than a convergence guarantee for an unspecified objective function. An optimization checkpoint needs these histories and the step count to reproduce subsequent steps. AdamW adds decoupled weight decay; feeding a penalty gradient into Adam instead implements coupled L2 regularization and changes the histories.

Adam bias correction compensates for the initial zero values in Adam's exponentially weighted gradient histories. The basic algorithm usesThese factors normalize the coefficient sums of the two running histories.^[ref]

For a scalar coordinate, suppose $m_0=0$ andExpanding the recurrence relation givesIf every $g_s$ has the same expectation $\mu$, linearity of expectation and the geometric series yieldThus ${\hat{m}}_t$ is an unbiased estimator of $\mu$ under these assumptions. Correlation between different times is allowed by this argument. Applying the same calculation to squared gradients with constant raw second moment $\nu =\mathbb{E}[g_s^2]$ gives $\mathbb{E}[v_t]=(1-{\beta }_2^t)\nu$ and $\mathbb{E}[{\hat{v}}_t]=\nu$.

For a constant supplied gradient of 2, choose ${\beta }_1=0.9$ and ${\beta }_2=0.99$. The first uncorrected histories are $m_1=0.2$ and $v_1=0.04$, while their corrected values are 2 and 4. The normalization removes the missing coefficient mass caused by initialization.

It does not make a changing history equal to the latest gradient. For an original deterministic example, let ${\beta }_1=1/2$, $g_1=1$ and $g_2=3$. Then $m_2=7/4$, sowhich differs from $g_2=3$. More generally, changing moment expectations produce a weighted average of past expectations. State the moment assumptions before interpreting “bias-corrected” as unbiased for a present distribution. The nonlinear parameter update built from the corrected histories is a separate object from either moment estimator.

AdamW combines the adaptive update of Adam with decoupled weight decay. It maintains exponentially weighted histories of the supplied data gradient and its componentwise square, while applying parameter shrinkage separately. Ilya Loshchilov and Frank Hutter presented the method in their work on decoupling decay from gradient-based regularization.^[ref]

For minimization without optional algorithm variants, initialize $m_0=v_0=0$. In the convention documented by PyTorch, step $t\ge 1$ usesThe product $\odot$ is a Hadamard product; square roots and division are also coordinatewise. Assume $0\le {\beta }_1,{\beta }_2<1$, $ϵ>0$ and the stated learning rate ${\eta }_t$. The decay coefficient is $\lambda$. This explicitly fixes epsilon placement and the decay convention.^[ref]

The denominators $1-{\beta }_1^t$ and $1-{\beta }_2^t$ perform Adam bias correction for zero-initialized histories. For a constant supplied scalar gradient $g$, the corresponding geometric sums give $m_t=(1-{\beta }_1^t)g$ and $v_t=(1-{\beta }_2^t)g^2$. Their corrected values are therefore $g$ and $g^2$. The squared-gradient quantity concerns a raw second moment, rather than a centered variance: a constant gradient of 2 has square 4 even though its variance is zero.

For an original first-step example, choose $f(\theta )={\theta }^2$, ${\theta }_0=1$, $\eta =0.1$, $\lambda =0.2$, ${\beta }_1=0.9$, ${\beta }_2=0.99$ and $ϵ=0.1$. ThenandThe constants are chosen for transparent arithmetic. Feeding $g_1+\lambda {\theta }_0=2.2$ into the same adaptive formulas instead, with no separate shrinkage, would give approximately $0.904348$. That coupled L2 regularization also changes the stored histories. Resuming AdamW therefore requires the relevant optimizer state and step count as well as the parameter values and settings.

Let $X_1,X_2,\dots$ be real independent and identically distributed random variables with expected value $\mu =\mathbb{E}[X_1]$ and finite, strictly positive variance ${\sigma }^2=\mathbb{E}[(X_1-\mu {)}^2]$. Write $S_n={\sum }_{j=1}^n{X}_j$ and ${\overline{X}}_n=S_n/n$ for their sum and sample mean. The central limit theorem statesPrecisely, for every real $z$,where $\Phi$ is the standard normal distribution function. The conclusion is convergence in distribution to the standard normal distribution. If the variance is zero, every observation equals $\mu$ almost surely; the displayed normalization is then undefined and there is no fluctuating error to describe.

The striking feature is that the observations need not have a normal distribution. They may be binary, bounded, or strongly skewed. Their detailed probability distribution disappears from this limit except through the expected value and variance. The subtraction removes predictable growth: $\mathbb{E}[S_n]=n\mu$. The division implements variance normalization: variance additivity for independent random variables gives $\mathrm{Var}(S_n)=n{\sigma }^2$, so every $Z_n$ has expected value zero and variance one. For nonzero $\mu$, an uncentered sum drifts; without rescaling its spread grows.

For a concrete example, let each $X_j$ have a Bernoulli distribution with success probability $p=0.2$. Then $K_n=S_n$ has a binomial distribution, with expected value $np$ and variance $np(1-p)$. Its exact probability mass function and its normalized count areHere $\left(\genfrac{}{}{0ex}{}{n}{k}\right)$ is a binomial coefficient. Individual trials are conspicuously skewed, yet their centered, rescaled counts approach the same standard normal distribution as sums of fair trials.

At $n=100$, $K_n$ has expected value $20$ and standard deviation $4$. The exact binomial distribution gives $\mathbb{P}(16\le K_n\le 24)=0.740141\dots$. A normal approximation with the continuity correction uses the interval $15.5$ to $24.5$:The small discrepancy is an approximation error, not an equality supplied by the central limit theorem.

The strong law of large numbers and weak law of large numbers locate the average: ${\overline{X}}_n\to \mu$, respectively by almost sure convergence and convergence in probability. The central limit theorem magnifies that shrinking error and describes its shape. In terms of the sample mean, the resulting normal approximation is $N(\mu ,{\sigma }^2/n)$, with standard error $\sigma /\sqrt{n}$. This explains why four times as many independent observations halve the standard error. When $\sigma$ is known, ${\overline{X}}_n\pm 1.96\sigma /\sqrt{n}$ has approximately $95\%$ coverage for $\mu$ at large $n$, giving a confidence interval. This coverage refers to repeated samples, not a guarantee for an individual interval.

A useful application is the Monte Carlo method: estimate a success probability by the proportion of successful simulated trials. Nicholas Metropolis and S. Ulam's original paper, "The Monte Carlo Method" (1949), illustrates this with games of solitaire and estimation of a region's volume by randomly selected points, and discusses why estimates require probabilistic error control. The strong law of large numbers justifies convergence of such proportions for independent repetitions; the central limit theorem supplies their large-sample error scale when $0<p<1$.^[ref]

There are three useful limits to this picture:

A historical original-paper reference is J. W. Lindeberg's "Eine neue Herleitung des Exponentialgesetzes in der Wahrscheinlichkeitsrechnung" (1922), published in Mathematische Zeitschrift 15, pp. 211–225. The Lindeberg condition and Lindeberg-Feller central limit theorem below describe a version for sums whose summands need not share a probability distribution.^[ref]

A brief proof outline uses the characteristic function of $Y=(X_1-\mu )/\sigma$. Its expected value is zero and its variance is one, so the finite second moment givesThe Taylor expansion of the complex exponential, with its remainder controlled by $Y^2$, and the dominated convergence theorem justify this step without a third-moment assumption. Independence then gives the characteristic function of $Z_n$:The limit is the characteristic function of the standard normal distribution; the Lévy continuity theorem turns this into convergence in distribution.

For a real random variable $X$ with finite variance ${\sigma }^2$, expected value $\mu$, and a threshold $a>0$, Chebyshev's inequality givesOne can also cap this upper bound at one, since a probability cannot exceed one. If $\sigma >0$, taking $a=k\sigma$ gives $\mathbb{P}(|X-\mu |\ge k\sigma )\le 1/k^2$. For example, at least $3/4$ of the probability lies strictly within two standard deviations of the mean. This statement needs only the finite variance assumption. If the variance is zero, $X=\mu$ almost surely and every positive deviation threshold has zero tail probability.

Consider a random variable uniformly distributed on $\{-2,-1,0,1,2\}$. Its expected value is zero and its variance is $(4+1+0+1+4)/5=2$. At $a=2$ the actual tail probability is $2/5$, while Chebyshev's inequality gives $2/2^2=1/2$. At $a=3$ the actual tail probability is zero, while the bound is $2/9$. A valid bound need not describe the exact shape or bounded support of a particular probability distribution.

For the standard normal distribution, the actual tail probabilities beyond two and three standard deviations are approximately $0.0455003$ and $0.0026998$, whereas the general bounds are $1/4$ and $1/9$. More knowledge of the probability distribution can therefore improve the estimate. The comparison does not diminish the guarantee: the three-point example in equality in Chebyshev's inequality attains it exactly.

The proof is a one-line use of Markov's inequality on the nonnegative random variable $Y=(X-\mu {)}^2$. The event $|X-\mu |\ge a$ equals $Y\ge a^2$, and $\mathbb{E}Y={\sigma }^2$. This explains why the denominator is squared and why a finite variance supplies the necessary information. ^[ref].

One important use is control of a sample mean. For independent and identically distributed random variables with finite variance ${\sigma }^2$, the variance of a sample mean is ${\sigma }^2/n$. HenceFor every fixed tolerance, the bound tends to zero as the sample size grows, proving the finite-variance case of the weak law of large numbers. For ${\sigma }^2=1$ and $\epsilon =0.1$, $n=2000$ makes the bound $0.05$. This is a sufficient bound under the stated assumptions, rather than an optimal required sample size.

An early historical source is Pafnuty Chebyshev's 1867 paper “Des valeurs moyennes”, published in French translation by N. de Khanikof. Its opening theorem bounds deviations of sums using their expected values and second moments, linking averages with control of exceptional outcomes. ^[ref].

Here $I_t={\sum }_{j=1}^t{1}_{\{{\xi }_j^{(1)}={\xi }_j^{(2)}\}}$ counts simultaneous collisions of two independent walks. A shared environment sign contributes $\mathbb{E}(1+\epsilon h{)}^2=1+{\epsilon }^2$; distinct signs contribute one. If the difference walk returns to zero with probability $\rho <1$, its total number of positive-time returns has law $\mathbb{P}(I_{\infty }=k)=(1-\rho ){\rho }^k$. Therefore the second moments are bounded by $(1-\rho )/(1-\rho (1+{\epsilon }^2))$ whenever $\rho (1+{\epsilon }^2)<1$. Convergence in L2 then preserves the limit's mean one.

For a centered complex Gaussian variable $Z$ whose real and imaginary parts are jointly Gaussian, the displayed formula follows by analytic continuation of the real joint moment-generating function. It uses the nonconjugated Second moment $\mathbb{E}[Z^2]$, not the positive variance $\mathbb{E}[|Z{|}^2]$. This distinction matters for the coherent mean in the Rytov approximation.

For a real-valued stochastic process $(X_t)$ with finite second moments, write $m(t)=\mathbb{E}{X}_t$. Its covariance function isThe diagonal gives the variance at each time; off-diagonal values record linear dependence between times. For the independent standard normal random variables above, $C_X(m,n)=1_{\{m=n\}}$, whereas the shared Gaussian process $Y_n=Z$ has $C_Y(m,n)=1$ for all $m,n$. For a fair simple random walk on the integer line, $C(m,n)=\min (m,n)$; Brownian motion has the continuous-time counterpart $C(s,t)=\min (s,t)$. These formulas quantify how much random history the observations share.

Every covariance function is a positive-semidefinite kernel. Indeed, for any real coefficients $a_1,\dots ,a_k$,Thus a proposed covariance function must make every finite covariance matrix positive semidefinite. For a Gaussian process, the mean function and the covariance function determine every finite-dimensional distribution, including degenerate ones such as the shared process. Conversely, any real positive-semidefinite kernel satisfying $C(s,t)=C(t,s)$, together with a specified mean function, defines consistent multivariate normal distributions; the Kolmogorov extension theorem then constructs a Gaussian process. For a general stochastic process, covariance alone does not determine the full law. The exceptional completeness of this description is what makes Gaussian process regression possible with a mean and a covariance model.

For a real random vector $X=(X_1,\dots ,X_p{)}^T$ with finite second moments, its covariance matrix collects all pairwise covariances:The diagonal entries are variances. The covariance matrix is a symmetric matrix and a positive semidefinite matrix, because for every real vector $a$,Thus one matrix describes the variance of every linear combination of the coordinates. It is a positive-definite matrix exactly when no nonzero linear combination is constant almost surely. A deterministic linear relation produces a zero eigenvalue.

For example,has unit coordinate variances, covariance $0.8$, and eigenvalues $1.8$ and $0.2$. Its unit eigenvectors $(1,1{)}^T/\sqrt{2}$ and $(1,-1{)}^T/\sqrt{2}$ describe directions of greatest and least projected variance. For a unit vector $a=(\cos \theta ,\sin \theta {)}^T$, direct multiplication gives $a^T\Sigma a=1+0.8\sin (2\theta )$: the variation in the sum direction is nine times that in the difference direction.

The spectral theorem supplies an orthonormal basis of these directions. This connects covariance matrices to principal component analysis: its leading eigenvector identifies the unit-direction projection with the largest variance. For a linear transformation $A$, the transformed random vector has covariance matrix $A\Sigma A^T$, so uncertainties propagate through a linear model by matrix multiplication.

For real random vectors $\mathbf{a}\in {\mathbb{R}}^p$ and $\mathbf{b}\in {\mathbb{R}}^q$ with finite second moments, their cross-covariance matrix collects the covariances between the two sets of coordinates:It has size $p\times q$, and exchanging the two vectors transposes it: $C_{ba}=C_{ab}^T$. Unlike a covariance matrix of a vector with itself, a cross-covariance matrix need not be square or symmetric.

For an original construction, let centered independent random variables $X,Z$ have variances1and4, and putLinearity of covariance givesThe second entry contains a negative contribution from $Z$, although the total is positive. Also $\mathrm{Var}(b)=13$ andThis joint covariance matrix is positive semidefinite. That restriction couples its diagonal and off-diagonal blocks; an arbitrary rectangular matrix cannot be combined with arbitrary within-vector covariances to produce a valid joint model.

For fixed real matrices $A,B$ and constant vectors $c,d$, direct substitution into the definition givesThis identity propagates shared input fluctuations through two different linear readouts, for example when assessing shared calibration error.

From $n$ independent identically distributed paired observations, the usual unbiased estimate isThe pairing must be preserved: independently shuffling the second set changes the estimated association. NIST's multivariate tutorial gives the coordinate sample-covariance formula and its assembly into a covariance matrix; the rectangular formula above is the block joining the two chosen coordinate groups.^[ref]

For jumps $r$ from state $x$ at nonnegative rates $W(x,r)$, the Fokker-Planck diffusion matrix in the infinitesimal-covariance convention isAssume the second jump moments are finite. Over a short interval $dt$, the conditional covariance of the state increment is $B(x)dt+o(dt)$. Subtracting the product of the mean increments changes only order-$d{t}^2$ terms. The matrix is symmetric and positive semidefinite because $v^{\mathsf{T}}Bv={\sum }_{r}W(x,r)(v\cdot r{)}^2\ge 0$ for every real vector $v$.

With this definition, its term in the Fokker-Planck equation is $\frac{1}{2}{\sum }_{i,j}{\partial }_i{\partial }_j(B_{ij}p)$. It is therefore $B/2$, not $B$, that corresponds to a scalar diffusion coefficient multiplying ${\partial }_x^{2}p$ in one dimension. Unit births and deaths at rates $b,d$ give drift $b-d$ and increment variance $B=b+d$: opposite mean displacements cancel in the drift but both contribute positively to the variance. The chemical diffusion matrix applies the same outer-product construction to reaction channels.

For $f(E,L)=f_E(E)/L$, with $L=rv\sin \eta$ and $E=\psi -v^2/2$, velocity integration gives $r\rho =2{\pi }^2{\int }_0^{\psi }{f}_E(E)dE$. Thus $f_E(E)=(2{\pi }^2{)}^{-1}d(r\rho )/d\psi$ at $\psi =E$. The factor $1/L$ is locally integrable under the velocity-volume measure. Its angular measure is uniform in $\eta$; averages of ${\cos }^2\eta$ and ${\sin }^2\eta$ are both $1/2$, so the total tangential second moment equals the radial second moment and the velocity-anisotropy parameter is $\beta =1/2$. At the largest allowed radius $\psi (r_E)=E$, the derivative becomes $[\rho +r(d\rho /dr)]/(d\psi /dr)$, not $[\rho +d\rho /d\psi ]/(d\psi /dr)$. Positivity requires $d(r\rho )/d\psi \ge 0$.

Jensen's inequality says that applying a convex function to a weighted mean cannot exceed the same weighted mean of its outputs. For nonnegative weights $w_i$ with ${\sum }_{i=1}^n{w}_i=1$ and inputs $x_i$ in a convex set,Nonnegative weights are essential: this is a convex combination, rather than an arbitrary affine combination. For two inputs it is precisely the defining inequality for a convex function; repeatedly grouping inputs gives the finite weighted version.

Take $f(x)=x^2$, weights $1/4$ and $3/4$, and inputs $1$ and $3$. Their weighted mean is $2.5$, so the two sides areThe Jensen gap is $0.75$. The inequality measures the effect of replacing distinct inputs by one representative input before applying a nonlinear function.

The probabilistic form uses an integrable random variable $X$ taking values in an interval where $f$ is a convex function. If $f(X)$ is also an integrable random variable and $\mathbb{E}X$ belongs to that interval, thenThe direction reverses for a concave function. For a strictly convex function, equality in the finite weighted form requires all inputs having positive weight to be equal; in the probabilistic form it requires $X$ to be constant almost surely. An affine function gives equality for any admissible distribution. ^[ref].

For the quadratic function $f(x)=x^2$ and a random variable with a finite second moment, the difference between the two sides is exactly its variance:Consider two equally likely values $2-d$ and $2+d$. Their expected value stays at $2$ as $d$ increases, but their variance is $d^2$. Thus the expected value of the square grows from $4$ to $4+d^2$. For the exponential function, the same two-point distribution gives $\mathbb{E}[e^X]=e^2\cosh d$, while $e^{\mathbb{E}X}=e^2$. Both are examples of Jensen's inequality, but their Jensen gaps depend differently on the spread of the inputs.

Another useful choice is the concave function $\log x$ on positive inputs. Its version of Jensen's inequality gives ${\sum }_i{w}_i\log x_i\le \log ({\sum }_i{w}_i{x}_i)$. Applying the increasing exponential function shows that the weighted geometric mean cannot exceed the weighted arithmetic mean. For the equally weighted inputs $1$ and $9$, these are $3$ and $5$. This recovers the arithmetic-geometric mean inequality and illustrates how choosing one function can generate an entire family of bounds.

A short proof idea explains the probabilistic form. If $f$ has a subgradient $g$ at $\mu =\mathbb{E}X$, the supporting affine function gives $f(X)\ge f(\mu )+g(X-\mu )$. Taking the expected value cancels the linear term because $\mathbb{E}[X-\mu ]=0$. This argument applies, for example, to finite convex functions on $\mathbb{R}$; more general domains of a function need attention to endpoints and the existence of a subgradient.

Johan Jensen's 1906 paper “Sur les fonctions convexes et les inégalités entre les valeurs moyennes” is an early source for the connection between convex functions and inequalities for means. ^[ref].

For independent zero-mean square-integrable increments, their partial sums form a martingale, and their squares form a nonnegative submartingale. Apply the Doob maximal inequality to the squared process at threshold $x^2$. Independence and zero means make its terminal second moment the sum of the increment variances. Identical distributions are not required.

Kriging predicts an unobserved value of a spatial random field by combining observations according to their covariances. Nearest-neighbor interpolation uses the closest sample; kriging also asks how strongly each sample is related to the target and how much information the samples share with one another. Two nearby, highly correlated samples need not provide twice the information of one. The different forms of kriging depend on whether the mean is known, an unknown constant, or an unknown regression drift.

Write $Z=(Z(s_1),\dots ,Z(s_n){)}^T$ for the observation random vector, $z$ for its realized values, and $Z_0=Z(s_0)$ for the prediction target. Assume finite second moments, and defineHere $\Sigma$ is the observation covariance matrix, assumed to be a positive-definite matrix in the formulas below. A covariance function supplies its entries and the target covariance vector $c$. For a second-order stationary random field, the covariogram $C(h)$ depends on displacement, and the semivariogram satisfies $\gamma (h)=C(0)-C(h)$. The intrinsically stationary random field formulation permits kriging using a semivariogram even when a stationary covariogram is unavailable; the finite covariance formulation below is a convenient important case.

For a linear combination $w^{T}Z$ predicting $Z_0$ with zero bias, the error variance equals its mean squared error:Ordinary kriging and universal kriging minimize this quadratic form subject to their mean constraints. Their optimality is among the specified linear predictors with zero bias under the chosen statistical model; covariance information alone does not establish optimality among arbitrary nonlinear predictors. The Gaussian random field interpretation supplies additional conclusions in simple kriging.

The reported error variance assumes a chosen covariance function, its statistical parameters, observation locations, and a mean structure. It omits uncertainty in fitted covariance parameters and does not validate the trend. A repeated-sampling mean squared error need not equal the error variance conditional on each realized data vector.^[ref]

In 1963, Georges Matheron presented kriging for estimating average ore grades in mining panels from weighted samples taken inside and outside a panel. This setting in geostatistics explains why prediction targets can be averages over regions as well as values at points.^[ref]

The law of total variance separates the spread of a population into spread inside groups and spread between their means. For a real square-integrable random variable $X$, so that $\mathbb{E}[X^2]<\infty$, and another random variable $Y$ on the same probability space,The first term averages the conditional variance over the possible information $Y$; the second measures how much the conditional mean changes with that information. There is no independence assumption, and $Y$ need not be numerical or have finite variance. Conditioning is useful precisely when knowing $Y$ tells us something about $X$. See MIT tutorial on using conditional expectation and variance.

More generally, the available information can be a sub-sigma-algebra $\mathcal{G}$ of the ambient probability space. With $m=\mathbb{E}[X\mid \mathcal{G}]$, the existing conditional expectation definition givesTaking $\mathcal{G}=\sigma (Y)$ recovers the displayed formula. The conditional mean and conditional variance are themselves random variables: before learning the group, we do not yet know which group average or group spread will apply. Their outer expectation and variance turn these varying quantities into numbers.

For a fully explicit mixture distribution, choose a group $Y=A$ or $Y=B$ with equal probabilities. In group $A$, choose $X=1$ or $X=3$ with equal conditional probabilities; in group $B$, choose $X=5$ or $X=11$ with equal conditional probabilities. Thus the complete joint probability distribution consists of $(A,1),(A,3),(B,5),(B,11)$, each with probability $1/4$, and no other outcomes. The conditional means are $2$ and $8$. Squaring the deviations $\pm 1$ and $\pm 3$ gives conditional variances $1$ and $9$, respectively. The law of total expectation gives the overall mean $\mu =(2+8)/2=5$.

The within-group contribution to variance isFor the between-group contribution, the conditional mean takes values $2$ and $8$, each three units from the overall mean. Consequently,We can check this without any conditioning: the discrete random variable $X$ takes values $1,3,5,11$ with equal probabilities, soEquivalently, its second moment is $(1+9+25+121)/4=39$, and subtracting the squared mean gives $39-25=14$. These are population variances, with the stated probabilities as weights, rather than estimates using the corrected denominator of a sample variance.

The figure makes the two sources of spread visible. The outcomes in group $A$ cluster tightly around $2$, while those in group $B$ sit farther apart around $8$. Replacing every outcome by its conditional mean removes the internal spread but leaves the separation between groups. Conversely, subtracting each group's conditional mean aligns the two groups around zero. Averaging only the conditional variances would miss nine units of variance; retaining only the conditional means would miss five. Neither summary alone describes the whole population.

Two modifications clarify what each contribution measures. Shift group $A$ to outcomes $4,6$ and group $B$ to outcomes $2,8$, keeping all four probabilities equal. Both conditional means are now $5$, while the conditional variances remain $1,9$. The between-group contribution vanishes, and total variance is just $5$. Group membership still tells us how uncertain an outcome is, even though the best point prediction stays the same. Alternatively, let every outcome in group $A$ equal $2$ and every outcome in group $B$ equal $8$. Now both conditional variances vanish, and all nine units of total variance come from group separation. Knowing the group predicts the outcome perfectly. These limiting cases distinguish differences in average outcome from differences in internal spread; the original mixture distribution contains both.

The same calculation answers a prediction question. If the group is hidden, the best constant prediction is the overall mean $5$, with mean squared error $14$, by variance as the minimum mean squared error of a constant. If the group is revealed, predict $2$ for $A$ and $8$ for $B$. The errors become $-1,1,-3,3$, and their mean squared error is $5$. Knowing the group therefore improves the best achievable mean squared error by $9$, exactly the between-group contribution. This is an interpretation relative to the information available: the remaining uncertainty could shrink further if additional information were revealed.

For any square-integrable measurable function $a(Y)$ used as a predictor, write $m(Y)=\mathbb{E}[X\mid Y]$. The prediction identity isThus predicting with the regression function $m(y)=\mathbb{E}[X\mid Y=y]$ gives the conditional expectation $m(Y)$ and minimizes mean squared error over every permitted predictor, including nonlinear ones. This minimizes average squared error, not the error for each individual outcome. See Berkeley Statistics 201A notes, sections 36–37.

Why do the two pieces add cleanly? Put $R=X-m(Y)$. Its conditional expectation given $Y$ is zero, so $\mathbb{E}[R]=0$ and $\mathbb{E}[Rb(Y)]=0$ for every square-integrable measurable function $b(Y)$. In particular, $R$ and $m(Y)$ are uncorrelated random variables. Splitting $X-\mu =R+(m(Y)-\mu )$ and squaring therefore leaves no cross term after taking expectation. Geometrically, the conditional expectation is an orthogonal projection in the L2 space, and the law of total variance is its Pythagorean theorem for centered random variables.

In our example, the four products $R(m(Y)-5)$ are $3,-3,-9,9$, which average to zero. Yet $R$ and $Y$ are not independent: observing $|R|=1$ identifies group $A$, while observing $|R|=3$ identifies group $B$. This is a concrete instance of heteroscedasticity, with different conditional variances despite zero conditional means of the errors. A group can predict uncertainty even when it cannot predict the error's sign.

The law of total variance is useful whenever aggregated data conceal distinct groups. Changing mixture weights changes both contributions, so the weighting must match the population being described. For example, giving group $A$ weight $3/4$ changes the overall mean to $7/2$, the within-group variance contribution to $3$, and the between-group contribution to $27/4$, making the total $39/4$. Group differences account for spread and prediction gains; assigning them a causal explanation requires a separate causal inference argument.

The Lévy-Itô decomposition builds a real Lévy process from drift, continuous Gaussian noise and Poisson jumps. For the triplet $(a,b,K)$ in the Lévy-Khintchine formula with small-jump cutoff one, take independent standard Brownian motion $B$ and Poisson random measure $N$ of intensity $dsK(dx)$. ThenHere $b\ge 0$, $K$ is a Lévy measure and $\tilde{N}=N-dsK(dx)$ denotes compensation. The large-jump term has finitely many jumps on each bounded time interval because $K(|x|>1)<\infty$.

The small-jump integral is defined by limits, rather than by subtracting two potentially infinite sums. For $\epsilon >0$, let $J_t^{\epsilon }={\int }_0^t{\int }_{\epsilon <|x|\le 1}x\tilde{N}(ds,dx)$. These are square-integrable martingales. The compensated Poisson isometry givesas $0<\delta <\epsilon \downarrow 0$. Doob's L2 maximal inequality also controls the supremum on $[0,T]$, so the limit exists locally uniformly in probability, indeed in the L2 norm of that supremum. This uses exactly the small-jump second-moment condition of a Lévy measure.^[ref]

The components have independent increments; multiplying their characteristic functions recovers the Lévy-Khintchine exponent. The decomposition separates continuous fluctuations from large jumps and possibly infinitely many small jumps. A finite small-jump Second moment does not require a finite first absolute jump moment, which explains why compensation and its limiting construction are essential.

A linear least-squares projection is the affine predictor that minimizes mean squared error. For real random variables $X,Y$ with finite second moments and $\mathrm{Var}(X)>0$, minimizing $\mathbb{E}(Y-a-bX{)}^2$ givesThe residual $R=Y-a-bX$ satisfies $\mathbb{E}R=0$ and $\mathbb{E}(RX)=0$. Consequently it is orthogonal, in the inner product $⟨U,V⟩=\mathbb{E}(UV)$, to the span of the constant variable and $X$. For any other affine predictor $a^{\prime }+b^{\prime }X$, expanding the square yieldsbecause the cross term vanishes. This proves optimality and uniqueness of the predicted random variable. The residual variance is the Schur complement covariance. No normal distribution is needed. For several predictors, the same orthogonality argument gives the normal equations for Linear least squares; a singular predictor covariance can make coefficients nonunique while leaving the projected prediction unique.

For a stable queue with finite long-run averages, a positive finite admitted rate, and negligible initial and terminal contributions after division by observation time, Little's law statesHere $L$ is the time-average queue population, ${\lambda }_{\mathrm{eff}}$ is the admitted rate through the chosen system, and $W$ is the customer-average customer sojourn time inside that same boundary. Without abandonment, admitted rate equals completed-service queue throughput in this stable regime. It may differ from the offered arrival rate because of blocking. When customers abandon, count their admissions and exits too, and average their actual time inside; using only service completions would mismatch the population being measured.

For the waiting area alone the matching identity iswhere $W_q$ is mean queueing delay. For one server with no abandonment, service in arrival order, and finite mean service time $\mathbb{E}S$,with $\rho$ the server utilization. Mixing total population $L$ with waiting-only duration $W_q$ would count different portions of each customer's visit. Customers who start service immediately contribute zero to $W_q$, and must still be included in that customer average.

The strength of Little's law is that it does not require a Poisson process, an exponential distribution of service times, or service in arrival order. The sample-path area identity for Little's law explains why: counting customer-time across time slices or across customer visits gives the same area. A sufficient probabilistic setting is repeated independent and identically distributed empty-to-empty cycles with finite positive mean cycle duration, finite positive mean customer count, and finite mean population area; the renewal-reward theorem then supplies the long-run averages. Stability alone does not guarantee finite mean queueing delay: the Pollaczek-Khinchine mean formula shows why a finite second moment of service time matters for that mean in an M/G/1 queue.

For example, suppose a service desk receives six requests per hour but admits only four; measured mean customer sojourn time of admitted requests is half an hour. Its mean queue population is $4\cdot (1/2)=2$ customers. Multiplying the offered rate by that duration would incorrectly give three. Conversely, counting two customers on average and measuring admitted queue throughput of four per hour gives $W=2/4=1/2$ hour. This lets one estimate a hard-to-track duration from occupancy and completed-job counts.

John D. C. Little published a rigorous proof in 1961 under finite-mean, strict stationarity, and ergodicity assumptions on the arrival process. Those were assumptions of that proof, rather than a requirement that every useful form of the identity have random arrivals. ^[ref]. ^[ref].

The Lyapunov moment inequality says that higher positive absolute moments give no smaller absolute moment sizes on a probability space. For a real random variable $Z$ and $0<r<s$,allowing an infinite value. For $r\ge 1$ the left expression is an Lp norm; below one it is a quasi-norm, so the usual triangle inequality terminology needs that distinction. The comparison uses a probability measure of total mass one; arbitrary unnormalized measure spaces need a factor depending on their total mass.

For a nonnegative count, the first moment size is its expected value, while the second is the square root of its second moment. Both ten-trial counts in second moment method have first moment size one. The independent count has second moment size $\sqrt{1.9}\approx 1.37840$, whereas the shared-outcome count $N=10B$ has second moment size $\sqrt{10}\approx 3.16228$. The second moment distinguishes models with the same average because its square increases the influence of rare large counts.

More generally, for $N=10B$ with a Bernoulli random variable $B$ of parameter $1/10$,It rises from one at $r=1$ toward ten as $r\to \infty$. Replacing $1/10$ by a parameter $0<q\le 1$ gives $10q^{1/r}$, which decreases toward zero for a fixed positive exponent when $q\downarrow 0$, making the rare-event and large-exponent limits different questions. For a scaled Rademacher random variable taking values $-2$ and two with equal probabilities, every absolute moment size is exactly two: increasing the exponent does not distinguish signs when the absolute value is constant.

For finite $\mathbb{E}|Z{|}^s$, apply the reversed Jensen's inequality to the strictly concave function $t^{r/s}$ and the nonnegative random variable $|Z{|}^s$:Taking the positive $r$th root proves the claim. Equality in this finite case holds exactly when $|Z|$ is constant almost surely. If the higher moment is infinite, the upper bound is infinite and remains valid. Exponent zero is excluded from the statement; logarithmic or geometric limiting summaries require separately stated conventions, especially when $Z$ can vanish.