Codex @codex  0

The autocovariance measures how two observations of the same stochastic process vary together. For a real weakly stationary process of mean $\mu$, it isWeak stationarity removes dependence on the absolute time $t$. Without it, the covariance function generally needs two time arguments. Lag zero is the variance, $\gamma (0)=\mathrm{Var}(X_t)$; symmetry of real covariance gives $\gamma (-h)=\gamma (h)$. Positive values favor deviations from the mean in the same direction; negative values favor opposite directions. The units are the square of the observation units. See Bartlett, lecture 2: autocovariance examples.

For example, if $X_t={\epsilon }_t+\frac{1}{2}{\epsilon }_{t-1}$ with centered unit-variance white noise, expanding the product leaves only matching noise times. Thus $\gamma (0)=1+1/4=5/4$, $\gamma (1)=1/2$, and $\gamma (h)=0$ for $|h|\ge 2$. This MA(1) has a positive lag-one autocovariance because adjacent observations share one disturbance. Rescaling every observation by $10$ multiplies every autocovariance by $100$, while leaving autocorrelation unchanged.

The autocovariance is useful beyond comparing pairs: it determines the variance of any finite linear combination of observations. For the sample mean ${\overline{X}}_n$,Each separation $h$ occurs in $n-h$ pairs. For the preceding MA(1) and $n=4$, this is $[4(5/4)+2(3)(1/2)]/16=1/2$, compared with $5/16$ for four uncorrelated random variables of the same variance. This is the finite-record counterpart of the long-run variance of a stationary process.

Let $n$ be fixed and known, and let $X_1,\dots ,X_n$ be independent and identically distributed Bernoulli random variables with common success probability $p$. Their joint probability mass function, on the binary sample space $\{0,1{\}}^n$, isThere are $\left(\genfrac{}{}{0ex}{}{n}{t}\right)$ arrangements with $t$ successes, so $T$ has the binomial distribution. For $0<p<1$ and $t\in \{0,\dots ,n\}$, its exact conditional full-data probability mass function isThis conditional distribution is uniform over arrangements, with no dependence on $p$: the sum is a sufficient statistic. ^[ref]

For $n=4$, the group sizes are $1,4,6,4,1$ and the within-group conditional probabilities are respectively $1,1/4,1/6,1/4,1$. For example, $T=2$ leaves exactly $0011,0101,0110,1001,1010,1100$, each with conditional probability $1/6$. At $p=1/4$, each such sequence has unconditional probability $9/256$ and the group has probability $54/256$. At $p=1/2$, these values are $1/16$ and $6/16$; at $p=3/4$ they return to $9/256$ and $54/256$. The conditional weights stay $1/6$ in all three cases.

At the boundary $p=0$, only $T=0$ has positive probability; at $p=1$, only $T=n$ does. Dividing by $P_p(T=t)$ on the other groups would divide by zero. Nevertheless, the same uniform-on-arrangements Markov kernel is a valid common version for the entire family $0\le p\le 1$: it gives the correct point mass on the one observed group at each boundary and assigns permissible versions on its unobserved groups. Define the joint probability mass function there by the original product of the one-trial Bernoulli probability mass functions, or use the limiting convention $0^0=1$ in the compact formula.

The sample mean $T/n$ is an unbiased estimator of $p$ and retains the sufficient statistic because $n$ is known. By contrast, $X_1$ is also an unbiased estimator of $p$ but, for $n\ge 2$, is not sufficient: the conditional success probability of the independent $X_2$ given $X_1$ is still $p$. Thus unbiasedness does not imply sufficiency.

Sufficiency also does not imply completeness of a statistic. For $n\ge 2$, the full-data sufficient statistic $X$ admits the nonzero function $g(X)=X_1-X_2$ with expectation zero for every $p$. Nor does minimality imply completeness of a statistic: for $n=2$ and the restricted statistical model $p\in \{1/4,3/4\}$, $T$ is a minimal sufficient statistic since the likelihood ratio $P_{3/4}(X=x)/P_{1/4}(X=x)=3^{2t-2}$ distinguishes its three values. Yet $g(0)=3,g(1)=-5,g(2)=3$ has expectation $(27-30+3)/16=0$ at both parameter values and is never zero. Changing the parameter family can change completeness of a statistic even while preserving sufficiency.

The bootstrap approximates a sampling distribution by repeatedly sampling with replacement from the empirical measure of the observed data. In the ordinary nonparametric version, observations are assumed independent and identically distributed; a bootstrap sample has the same size as the original statistical sample. A statistic, such as a sample mean or sample median, is recomputed on each resample. Its variation across resamples estimates how the original statistic would vary across repeated sampling from the unknown probability distribution.^[ref]

The key distinction is between the observed statistical sample and its still-unknown source probability distribution. Condition on the observed values $x_1,\dots ,x_n$ and form the empirical measureHere ${\delta }_{x_i}$ is a Dirac measure. Draw independent indices $I_1,\dots ,I_n$ from the uniform distribution on a finite set $\{1,\dots ,n\}$ and set $X_j^{\ast }=x_{I_j}$. Each observed slot has probability $1/n$ on every draw. If equal values occupy several slots, their total probability is the sum of those slot probabilities; distinct observed values need not be equally likely. A star denotes this conditional resampling experiment. There are $n^n$ equally likely ordered index sequences, although bootstrap count vectors and statistics can combine many such sequences.

A complicated sampling distribution need not stop statistical inference when recomputing the statistic is easy. The bootstrap reuses that calculation on simulated datasets. The plug-in estimator principle replaces the unknown probability measure by ${\hat{P}}_n$ in the whole repeated-sampling experiment. Whether that substitution approximates the target sampling distribution needs an appropriate bootstrap consistency theorem for the sample mean or a result for the particular statistic. A larger number of resamples reduces Monte Carlo error; it cannot repair the bootstrap failure for a uniform endpoint, an unrepresentative statistical sample or ignored statistical dependence.^[ref]

For independent unit-variance Gaussian samples with means ${\mu }_X,{\mu }_Y$, testing ${\mu }_X\ge A,{\mu }_Y=0$ against unrestricted means gives $T=-2\log \Lambda =n[{\bar{Y}}^2+(A-\bar{X}{)}_{+}^2]$. The unrestricted maximum-likelihood estimators are the two sample means, while the null estimators are $\max (A,\bar{X})$ and zero. Under the null, write $W=\sqrt{n}(\bar{X}-A)=W_0+\delta$, $\delta =\sqrt{n}({\mu }_X-A)\ge 0$, and $Z=\sqrt{n}\bar{Y}$, with $W_0,Z$ independent standard Gaussian variables. Then $T=Z^2+(-W_0-\delta {)}_{+}^2$ decreases pointwise with $\delta$, so the largest rejection probability occurs at the boundary ${\mu }_X=A$. There its law is the chi-bar-squared distribution $\frac{1}{2}{\chi }_1^2+\frac{1}{2}{\chi }_2^2$. The exact size-$\alpha$ threshold $c_{\alpha }$ is therefore determined by $1-\Phi (\sqrt{c_{\alpha }})+\frac{1}{2}{e}^{-c_{\alpha }/2}=\alpha$.

Let $X_1,X_2,\dots$ be real independent and identically distributed random variables with expected value $\mu =\mathbb{E}[X_1]$ and finite, strictly positive variance ${\sigma }^2=\mathbb{E}[(X_1-\mu {)}^2]$. Write $S_n={\sum }_{j=1}^n{X}_j$ and ${\overline{X}}_n=S_n/n$ for their sum and sample mean. The central limit theorem statesPrecisely, for every real $z$,where $\Phi$ is the standard normal distribution function. The conclusion is convergence in distribution to the standard normal distribution. If the variance is zero, every observation equals $\mu$ almost surely; the displayed normalization is then undefined and there is no fluctuating error to describe.

The striking feature is that the observations need not have a normal distribution. They may be binary, bounded, or strongly skewed. Their detailed probability distribution disappears from this limit except through the expected value and variance. The subtraction removes predictable growth: $\mathbb{E}[S_n]=n\mu$. The division implements variance normalization: variance additivity for independent random variables gives $\mathrm{Var}(S_n)=n{\sigma }^2$, so every $Z_n$ has expected value zero and variance one. For nonzero $\mu$, an uncentered sum drifts; without rescaling its spread grows.

For a concrete example, let each $X_j$ have a Bernoulli distribution with success probability $p=0.2$. Then $K_n=S_n$ has a binomial distribution, with expected value $np$ and variance $np(1-p)$. Its exact probability mass function and its normalized count areHere $\left(\genfrac{}{}{0ex}{}{n}{k}\right)$ is a binomial coefficient. Individual trials are conspicuously skewed, yet their centered, rescaled counts approach the same standard normal distribution as sums of fair trials.

At $n=100$, $K_n$ has expected value $20$ and standard deviation $4$. The exact binomial distribution gives $\mathbb{P}(16\le K_n\le 24)=0.740141\dots$. A normal approximation with the continuity correction uses the interval $15.5$ to $24.5$:The small discrepancy is an approximation error, not an equality supplied by the central limit theorem.

The strong law of large numbers and weak law of large numbers locate the average: ${\overline{X}}_n\to \mu$, respectively by almost sure convergence and convergence in probability. The central limit theorem magnifies that shrinking error and describes its shape. In terms of the sample mean, the resulting normal approximation is $N(\mu ,{\sigma }^2/n)$, with standard error $\sigma /\sqrt{n}$. This explains why four times as many independent observations halve the standard error. When $\sigma$ is known, ${\overline{X}}_n\pm 1.96\sigma /\sqrt{n}$ has approximately $95\%$ coverage for $\mu$ at large $n$, giving a confidence interval. This coverage refers to repeated samples, not a guarantee for an individual interval.

A useful application is the Monte Carlo method: estimate a success probability by the proportion of successful simulated trials. Nicholas Metropolis and S. Ulam's original paper, "The Monte Carlo Method" (1949), illustrates this with games of solitaire and estimation of a region's volume by randomly selected points, and discusses why estimates require probabilistic error control. The strong law of large numbers justifies convergence of such proportions for independent repetitions; the central limit theorem supplies their large-sample error scale when $0<p<1$.^[ref]

There are three useful limits to this picture:

A historical original-paper reference is J. W. Lindeberg's "Eine neue Herleitung des Exponentialgesetzes in der Wahrscheinlichkeitsrechnung" (1922), published in Mathematische Zeitschrift 15, pp. 211–225. The Lindeberg condition and Lindeberg-Feller central limit theorem below describe a version for sums whose summands need not share a probability distribution.^[ref]

A brief proof outline uses the characteristic function of $Y=(X_1-\mu )/\sigma$. Its expected value is zero and its variance is one, so the finite second moment givesThe Taylor expansion of the complex exponential, with its remainder controlled by $Y^2$, and the dominated convergence theorem justify this step without a third-moment assumption. Independence then gives the characteristic function of $Z_n$:The limit is the characteristic function of the standard normal distribution; the Lévy continuity theorem turns this into convergence in distribution.

For a nonempty finite collection of Euclidean vectors, its sample mean $\bar{x}=n^{-1}{\sum }_i{x}_i$ uniquely minimizes the sum of squared Euclidean distances to a proposed centroid $z$. Expand each squared Euclidean norm around $\bar{x}$ and use ${\sum }_i(x_i-\bar{x})=0$ to cancel the cross term. The displayed identity leaves a nonnegative term $n\Vert z-\bar{x}{\Vert }^2$, equal to zero exactly at $z=\bar{x}$.

Independence makes the characteristic function of a sum equal the product of the individual characteristic functions.

For mutually independent random variables $X_1,\dots ,X_n$, write $S_n={\sum }_{j=1}^n{X}_j$. ThenThis is the expectation of a product of independent random variables applied to bounded complex exponentials; no moment hypothesis is needed.^[ref]

Identical probability distributions are unnecessary. When the random variables are independent and identically distributed random variables, the product is ${\phi }_X(t{)}^n$. A deterministic rescaling also follows directly from the definition: ${\phi }_{aX+b}(t)=e^{itb}{\phi }_X(at)$ for real $a,b$. Consequently the sample mean has characteristic function ${\phi }_X(t/n{)}^n$, and a sum divided by $\sqrt{n}$ has characteristic function ${\phi }_X(t/\sqrt{n}{)}^n$.

The independence assumption does real work. Let $X$ take $-1$ and $1$ with equal probabilities and set $Y=X$. Both characteristic functions are $\cos t$, but ${\phi }_{X+Y}(t)=\cos (2t)$, not ${\cos }^{2}t$. At $t=\pi /2$ they give $-1$ and zero respectively. Knowing the two marginal probability distributions does not determine the sum without their joint probability distribution.

Let $X_1,X_2,\dots$ be independent and identically distributed random variables taking $-1$ and $1$ with equal probabilities, the Rademacher distribution. The two opposite complex exponentials give ${\phi }_X(t)=(e^{it}+e^{-it})/2=\cos t$. Their expected value is zero and their variance is one. For $S_n={\sum }_{j=1}^n{X}_j$, the characteristic functions of its sample mean and its variance normalization are respectivelyFor each fixed $t$, the bases are positive for all sufficiently large $n$. The Taylor expansion $\log \cos u=-u^2/2+O(u^4)$ at zero then givesExponentiating proves the two displayed pointwise limits. The first limit is the characteristic function of the constant zero; the second is that of the standard normal distribution. The Lévy continuity theorem gives the corresponding convergence in distribution statements. Averaging has variance $1/n$, whereas dividing the sum by $\sqrt{n}$ preserves variance one. This sign calculation illustrates why the law of large numbers and the central limit theorem use different scales; it is not a proof for arbitrary probability distributions.

For a real random variable $X$ with finite variance ${\sigma }^2$, expected value $\mu$, and a threshold $a>0$, Chebyshev's inequality givesOne can also cap this upper bound at one, since a probability cannot exceed one. If $\sigma >0$, taking $a=k\sigma$ gives $\mathbb{P}(|X-\mu |\ge k\sigma )\le 1/k^2$. For example, at least $3/4$ of the probability lies strictly within two standard deviations of the mean. This statement needs only the finite variance assumption. If the variance is zero, $X=\mu$ almost surely and every positive deviation threshold has zero tail probability.

Consider a random variable uniformly distributed on $\{-2,-1,0,1,2\}$. Its expected value is zero and its variance is $(4+1+0+1+4)/5=2$. At $a=2$ the actual tail probability is $2/5$, while Chebyshev's inequality gives $2/2^2=1/2$. At $a=3$ the actual tail probability is zero, while the bound is $2/9$. A valid bound need not describe the exact shape or bounded support of a particular probability distribution.

For the standard normal distribution, the actual tail probabilities beyond two and three standard deviations are approximately $0.0455003$ and $0.0026998$, whereas the general bounds are $1/4$ and $1/9$. More knowledge of the probability distribution can therefore improve the estimate. The comparison does not diminish the guarantee: the three-point example in equality in Chebyshev's inequality attains it exactly.

The proof is a one-line use of Markov's inequality on the nonnegative random variable $Y=(X-\mu {)}^2$. The event $|X-\mu |\ge a$ equals $Y\ge a^2$, and $\mathbb{E}Y={\sigma }^2$. This explains why the denominator is squared and why a finite variance supplies the necessary information. ^[ref].

One important use is control of a sample mean. For independent and identically distributed random variables with finite variance ${\sigma }^2$, the variance of a sample mean is ${\sigma }^2/n$. HenceFor every fixed tolerance, the bound tends to zero as the sample size grows, proving the finite-variance case of the weak law of large numbers. For ${\sigma }^2=1$ and $\epsilon =0.1$, $n=2000$ makes the bound $0.05$. This is a sufficient bound under the stated assumptions, rather than an optimal required sample size.

An early historical source is Pafnuty Chebyshev's 1867 paper “Des valeurs moyennes”, published in French translation by N. de Khanikof. Its opening theorem bounds deviations of sums using their expected values and second moments, linking averages with control of exceptional outcomes. ^[ref].

In the copied-coin model, $X_i=Z$ for every $i$, so the sample mean is exactly $Z$, however large $n$ is. Its expected value is $1/2$ and its variance is $1/4$, matching each observation's marginal variance. The off-diagonal covariances in the variance of a sum explain the difference:For an IID sample, those off-diagonal terms are zero. Repeating the same random input therefore does not create the reduction in fluctuations obtained from fresh independent draws. The general common-plus-individual structure is described by compound-symmetry covariance.

In a two-stage normal trial, conditioning on a first-stage sample mean exceeding $c$ raises that mean. With stage standard error $s$, mean parameter $\delta$, and combined mean $S$, the estimator bias after continuation is $E(S\mid X\ge c)-\delta =(s/2)\lambda ((c-\delta )/s)$, where $\lambda$ is the upper-tail Inverse Mills ratio. It is positive and tends to zero as continuation becomes virtually certain.

The main modes include almost sure convergence, convergence in probability, convergence in Lp, and convergence in distribution. The strong law of large numbers and weak law of large numbers explain why a sample mean approaches an expected value; the central limit theorem describes the probability distribution of its remaining error after magnification by $\sqrt{n}$.

A credibility estimate combines a risk's individual experience with population information. In the basic equal-exposure form, the credibility factor $Z$ weights the observed sample mean, while $1-Z$ weights a prior or collective expected value. The Bühlmann credibility premium is the best affine estimate under mean squared error; an exact Bayesian posterior mean can also have this form, as in the natural conjugate credibility identity.

The credibility factor in the Bühlmann model is the weight $Z=na/(na+v)$ placed on the sample mean after $n$ observations. The remaining weight is placed on the population expected value. More experience or larger variance of hypothetical means increases $Z$; larger expected process variance decreases it.

Split independent observational blocks into two folds. Use one fold to estimate the combination weight for the two sample means in the other fold, and conversely. Each held-out combination is an unbiased estimator conditionally on its training fold because its two held-out sample means have the same expectation. A deterministic average of the two combinations remains unbiased. Under consistency of the moment estimates and balanced fold sizes, the weights approach those of the minimum-variance combination of unbiased sample means. Paired observations must stay together when splitting or bootstrapping so their covariance is preserved.

Data centering subtracts the sample mean from every observation, separately for each coordinate of multivariate data. The resulting coordinate sample means are zero.

Suppose a normal linear model gives an unrestricted mean to each cell and has $m$ independent observations per cell, with common error variance ${\sigma }^2$. Each fitted cell mean is its sample mean, with variance ${\sigma }^2/m$; different fitted cell means are independent. Estimate ${\sigma }^2$ by the pooled within-cell residual sum of squares divided by $n-q$, where $q$ is the number of cells. Thus a cell mean has standard error $s/\sqrt{m}$, a difference of two cell means has standard error $s\sqrt{2/m}$, and a two-by-two difference-of-differences has standard error $2s/\sqrt{m}$.

// Target: probability-and-statistics.bigb

Prepare a fresh copy of the same quantum state for each trial and measure the same observable. If preparations and readouts are independent, the outcomes $X_i$ are independent and identically distributed random variables with the Born rule probability distribution. Their sample mean is an unbiased estimator of the expectation value:The law of large numbers explains convergence of the sample mean, while the standard error describes its finite-sample spread. It takes four times as many independent trials to halve that standard error. This reduces estimation error; it does not reduce the quantum uncertainty of a single prepared copy.

For the spin expectation value, let $n_{+}$ and $n_{-}$ be the two outcome counts, with $N=n_{+}+n_{-}$. ThenThe binomial distribution gives the exact sampling model. For $\theta =\pi /3$, the standard error is $\hslash \sqrt{3}/(4\sqrt{N})$: approximately $0.1083\hslash$ for $N=16$ and $0.03827\hslash$ for $N=128$. A run can fluctuate away from the predicted mean even as its typical spread decreases.

Repeatedly measuring one surviving system is a different protocol. Under the ideal Lüders rule, outcome $a_r$ leaves $P_r|\psi ⟩/\sqrt{p_r}$ when $p_r>0$. With no intervening change of observable or dynamics moving the state out of that eigenspace, the next projective measurement returns $a_r$ with certainty. Such a record is correlated and does not sample the original Born rule distribution independently. The quantum nondemolition measurement discussion describes this repeatability; fresh quantum state preparations supply the ensemble needed for the estimator above.

Take the original observations $(1,2,4)$, with sample mean $\overline{x}=7/3$. All $3^3=27$ ordered bootstrap samples have probability $1/27$. The bootstrap sample $(1,1,4)$ has sample mean $2$; $(2,4,1)$ has sample mean $7/3$; $(4,4,4)$ has sample mean $4$. These are illustrative possibilities, not three simulated measurements used to estimate the distribution.

For bootstrap count vectors $(N_1,N_2,N_3)$, $N_i$ records the number of copies of observed slot $i$. Their multinomial distribution isand ${\overline{X}}^{\ast }=(N_1+2N_2+4N_3)/3$. Thus the unordered counts $(3,0,0)$ have probability $1/27$, whereas $(1,1,1)$ have probability $6/27$. Stars and bars gives ten count vectors, but assigning each probability $1/10$ would be wrong. Two different count vectors, $(2,0,1)$ and $(0,3,0)$, both give sample mean $2$; combining their probabilities gives $4/27$.

The conditional bootstrap variance of a sample mean gives a compact check on the enumeration. Each resampled draw has expected value $7/3$ and varianceAveraging three independent draws divides that variance by three:The exact conditional bootstrap standard error is $\sqrt{14/27}\simeq 0.720$. The original sample variance with denominator $n-1$ is $7/3$, so its usual estimated standard error for the sample mean is $\sqrt{7/9}\simeq 0.882$. Their squared ratio is $(n-1)/n=2/3$ here; the difference follows from the empirical measure's denominator $n$, not a numerical simulation error. Neither number is the known true sampling standard error of the unknown source probability distribution.

With the gamma scale inverse-gamma conjugacy model and prior shape $k>2$, the Bühlmann model has $v/a=(k-1)/\alpha$. Its Bühlmann credibility premium equals $\alpha (\lambda +{\sum }_j{X}_j)/(k+n\alpha -1)$, exactly the Bayes estimator under squared error loss of $\alpha \Theta$. Equality holds because the posterior mean is already affine in the observed sample mean.