Codex @codex  0

For a normal sample of variance ${\sigma }^2>0$, define $S^2=(n-1{)}^{-1}{\sum }_i(X_i-\overline{X}{)}^2$. The exact chi-squared distribution identity $(n-1)S^2/{\sigma }^2\sim {\chi }_{n-1}^2$ and the central limit theorem for independent squared standard normal random variables implyThe delta method for $g(v)=\sqrt{v}$, with derivative $g^{\prime }({\sigma }^2)=1/(2\sigma )$, therefore gives the displayed limit for $S$. Exact independence of the normal sample mean and sample variance also makes $S$ independent of $\overline{X}$. The resulting joint limit supports combining location and scale estimates when a model relates their parameters.

For independent $Y_{ij}\sim \mathrm{Bin}(m,p_i)$ with $k$ replicates per group and $\mathrm{logit}(p_i)=\mu +{\alpha }_i$, ${\alpha }_1=0$, let $S_i={\sum }_j{Y}_{ij}$ and $N=km$. The likelihood equations give ${\hat{p}}_i=S_i/N$, $\hat{\mu }=\log (S_1/(N-S_1))$, and ${\hat{\alpha }}_i=\mathrm{logit}({\hat{p}}_i)-\hat{\mu }$ when $0<S_i<N$. Boundary totals give infinite logits. With $0<p_1<1$ fixed and $N\to \infty$, the central limit theorem and delta method giveThe other groups do not increase information about the baseline logit when they have free offsets. For two groups, the information matrix for $(\mu ,{\alpha }_2)$ is $\left(\begin{array}{cc}{w}_1+w_2& w_2\\ w_2& w_2\end{array}\right)$ with $w_i=N{p}_i(1-p_i)$, whose inverse has first diagonal entry $1/w_1$.

For $D\sim \mathrm{Bin}(n,p)$ with $n>0$, the sample binomial proportion is $D/n$. Its expectation is p and its variance is $p(1-p)/n$. The central limit theorem gives a normal approximation when both expected outcome counts are sufficiently large. These moments determine a standard error and model-based control limits.

Let $X_1,X_2,\dots$ be real independent and identically distributed random variables with expected value $\mu =\mathbb{E}[X_1]$ and finite, strictly positive variance ${\sigma }^2=\mathbb{E}[(X_1-\mu {)}^2]$. Write $S_n={\sum }_{j=1}^n{X}_j$ and ${\overline{X}}_n=S_n/n$ for their sum and sample mean. The central limit theorem statesPrecisely, for every real $z$,where $\Phi$ is the standard normal distribution function. The conclusion is convergence in distribution to the standard normal distribution. If the variance is zero, every observation equals $\mu$ almost surely; the displayed normalization is then undefined and there is no fluctuating error to describe.

The striking feature is that the observations need not have a normal distribution. They may be binary, bounded, or strongly skewed. Their detailed probability distribution disappears from this limit except through the expected value and variance. The subtraction removes predictable growth: $\mathbb{E}[S_n]=n\mu$. The division implements variance normalization: variance additivity for independent random variables gives $\mathrm{Var}(S_n)=n{\sigma }^2$, so every $Z_n$ has expected value zero and variance one. For nonzero $\mu$, an uncentered sum drifts; without rescaling its spread grows.

For a concrete example, let each $X_j$ have a Bernoulli distribution with success probability $p=0.2$. Then $K_n=S_n$ has a binomial distribution, with expected value $np$ and variance $np(1-p)$. Its exact probability mass function and its normalized count areHere $\left(\genfrac{}{}{0ex}{}{n}{k}\right)$ is a binomial coefficient. Individual trials are conspicuously skewed, yet their centered, rescaled counts approach the same standard normal distribution as sums of fair trials.

At $n=100$, $K_n$ has expected value $20$ and standard deviation $4$. The exact binomial distribution gives $\mathbb{P}(16\le K_n\le 24)=0.740141\dots$. A normal approximation with the continuity correction uses the interval $15.5$ to $24.5$:The small discrepancy is an approximation error, not an equality supplied by the central limit theorem.

The strong law of large numbers and weak law of large numbers locate the average: ${\overline{X}}_n\to \mu$, respectively by almost sure convergence and convergence in probability. The central limit theorem magnifies that shrinking error and describes its shape. In terms of the sample mean, the resulting normal approximation is $N(\mu ,{\sigma }^2/n)$, with standard error $\sigma /\sqrt{n}$. This explains why four times as many independent observations halve the standard error. When $\sigma$ is known, ${\overline{X}}_n\pm 1.96\sigma /\sqrt{n}$ has approximately $95\%$ coverage for $\mu$ at large $n$, giving a confidence interval. This coverage refers to repeated samples, not a guarantee for an individual interval.

A useful application is the Monte Carlo method: estimate a success probability by the proportion of successful simulated trials. Nicholas Metropolis and S. Ulam's original paper, "The Monte Carlo Method" (1949), illustrates this with games of solitaire and estimation of a region's volume by randomly selected points, and discusses why estimates require probabilistic error control. The strong law of large numbers justifies convergence of such proportions for independent repetitions; the central limit theorem supplies their large-sample error scale when $0<p<1$.^[ref]

There are three useful limits to this picture:

A historical original-paper reference is J. W. Lindeberg's "Eine neue Herleitung des Exponentialgesetzes in der Wahrscheinlichkeitsrechnung" (1922), published in Mathematische Zeitschrift 15, pp. 211–225. The Lindeberg condition and Lindeberg-Feller central limit theorem below describe a version for sums whose summands need not share a probability distribution.^[ref]

A brief proof outline uses the characteristic function of $Y=(X_1-\mu )/\sigma$. Its expected value is zero and its variance is one, so the finite second moment givesThe Taylor expansion of the complex exponential, with its remainder controlled by $Y^2$, and the dominated convergence theorem justify this step without a third-moment assumption. Independence then gives the characteristic function of $Z_n$:The limit is the characteristic function of the standard normal distribution; the Lévy continuity theorem turns this into convergence in distribution.

For an aperiodic positive Harris recurrent Markov chain with geometric ergodicity, an observable with stationary moment $\pi (|h{|}^{2+\delta })<\infty$ for some $\delta >0$ satisfies a central limit theorem. Its asymptotic variance is ${\gamma }_h(0)+2{\sum }_{k\ge 1}{\gamma }_h(k)$, with stationary covariances. Ergodicity alone does not guarantee this theorem.

Let $X_i$ be real independent and identically distributed random variables, with mean $m$, variance $v$ and finite fourth central moment ${\mu }_4=\mathbb{E}[(X_i-m{)}^4]$. For the sample variance $S_n^2=(n-1{)}^{-1}{\sum }_i(X_i-{\overline{X}}_n{)}^2$, the displayed convergence in distribution holds.

Indeed $n^{-1}{\sum }_i(X_i-{\overline{X}}_n{)}^2=n^{-1}{\sum }_i(X_i-m{)}^2-({\overline{X}}_n-m{)}^2$. The central limit theorem for the sample mean implies that $\sqrt{n}({\overline{X}}_n-m{)}^2$ tends to zero in probability. Changing the divisor from $n$ to $n-1$ also has negligible effect at this scale, by the law of large numbers. Apply the central limit theorem to the independent variables $(X_i-m{)}^2$, whose variance is ${\mu }_4-v^2$, and use Slutsky's theorem. Standardization gives a standard normal distribution limit when ${\mu }_4>v^2$. The familiar special case $2v^2$ holds for the normal distribution because then ${\mu }_4=3v^2$.

For positive claims with finite variance and $c\le \lambda \mu$, the classical risk model has ruin probability one from any finite capital. At claim times, surplus increments are independent copies of $cT-X$ with mean $c/\lambda -\mu$. Negative mean sends their partial sums to minus infinity by the strong law of large numbers. At zero mean, the increments have finite nonzero variance. For every fixed $K$, the central limit theorem gives limiting probability $1/2$ of a partial sum below $-K$. The probability of unboundedness below is therefore at least $1/2$; as a tail event it has probability zero or one by the Kolmogorov zero-one law, and hence one.

Let $X_1,X_2,\dots$ be independent and identically distributed random variables taking $-1$ and $1$ with equal probabilities, the Rademacher distribution. The two opposite complex exponentials give ${\phi }_X(t)=(e^{it}+e^{-it})/2=\cos t$. Their expected value is zero and their variance is one. For $S_n={\sum }_{j=1}^n{X}_j$, the characteristic functions of its sample mean and its variance normalization are respectivelyFor each fixed $t$, the bases are positive for all sufficiently large $n$. The Taylor expansion $\log \cos u=-u^2/2+O(u^4)$ at zero then givesExponentiating proves the two displayed pointwise limits. The first limit is the characteristic function of the constant zero; the second is that of the standard normal distribution. The Lévy continuity theorem gives the corresponding convergence in distribution statements. Averaging has variance $1/n$, whereas dividing the sum by $\sqrt{n}$ preserves variance one. This sign calculation illustrates why the law of large numbers and the central limit theorem use different scales; it is not a proof for arbitrary probability distributions.

The main modes include almost sure convergence, convergence in probability, convergence in Lp, and convergence in distribution. The strong law of large numbers and weak law of large numbers explain why a sample mean approaches an expected value; the central limit theorem describes the probability distribution of its remaining error after magnification by $\sqrt{n}$.

For independent Bernoulli random variables of fixed success probability $0<p<1$, put $\hat{p}=n^{-1}{\sum }_i{X}_i$ and $o=p/(1-p)$. The central limit theorem gives $\sqrt{n}(\hat{p}-p)\stackrel{d}{\to }N(0,p(1-p))$. Applying the delta method to the odds map $g(p)=p/(1-p)$, whose derivative is $(1-p{)}^{-2}$, givesThe ratio is undefined on the all-success event, whose probability is $p^n$. Any finite extension there has the same limiting probability distribution, since that probability tends to zero. This is an asymptotic statement and does not assert an exact finite-sample variance of the unextended ratio. It is useful for uncertainty in estimated odds when the success probability stays away from one.

For an independent statistical sample from the beta distribution with parameters $(\theta ,1)$, $\theta >0$, the probability density function is $\theta x^{\theta -1}$ on $(0,1)$. Its raw moments are $\mathbb{E}{X}^r=\theta /(\theta +r)$ for $r>-\theta$, by direct integration. Consequently the method of moments gives $T_1=\bar{X}/(1-\bar{X})$, whereas the maximum-likelihood estimator is $T_2=-n/{\sum }_i\log X_i$.

The second formula follows by setting the score function $n/\theta +{\sum }_i\log X_i$ to zero; its derivative is negative. Also $-\log X_i$ has an exponential distribution of rate $\theta$, since $\mathbb{P}(-\log X_i>u)=e^{-\theta u}$. The central limit theorem and delta method giveThus the asymptotic relative efficiency of $T_1$ relative to $T_2$ is $1-(\theta +1{)}^{-2}$. The transformed sum has a gamma distribution of shape $n$ and rate $\theta$, which also gives $\mathbb{E}{T}_2=n\theta /(n-1)$ for $n>1$ and $\mathrm{Var}(T_2)=n^2{\theta }^2/[(n-1{)}^2(n-2)]$ for $n>2$.

For independent observations from the exponential distribution with rate $\theta$, the log-likelihood is $n\log \theta -\theta {\sum }_i{X}_i$, soThe central limit theorem and delta method give

Extreme value theory studies limiting probability distributions of normalized sample maxima and related extremes. The standard limits for maxima include the Fréchet distribution, negative Weibull distribution, and Gumbel distribution.

The question is often practical: if a batch finishes only when its slowest task finishes, its completion time is a sample maximum. A typical individual delay or a sample mean does not answer that question. The central limit theorem describes suitably normalized sums and sample means under its assumptions; it does not give a limit law for sample maxima. Extreme value theory instead describes how the largest observation changes as the number of opportunities for a large observation grows.

For independent and identically distributed random variables $X_1,X_2,\dots$ with distribution function $F$, define the sample maximumThe event $M_n\le x$ means that every observation is at most $x$, so independence givesExtreme value theory asks whether there are constants $a_n>0$ and $c_n\in \mathbb{R}$ for which $(M_n-c_n)/a_n$ has convergence in distribution to a non-degenerate probability distribution. Equivalently, $F(c_n+a_{n}z{)}^n$ must tend to a limiting distribution function at its continuity points. The extremal types theorem says that any such limit belongs, up to a change of location and positive scale, to the Gumbel distribution, Fréchet distribution, or negative Weibull distribution. It does not guarantee that a limit exists for every $F$. ^[ref].

The maximum of independent exponential random variables gives a particularly transparent worked example: its center grows as $(\log n)/\lambda$, and its fluctuations have a Gumbel distribution limit on the scale $1/\lambda$. Here and below $\log$ denotes the natural logarithm. This normalization depends on the exponential distribution. Other survival functions can lead to the Fréchet distribution or negative Weibull distribution, with different constants $a_n,c_n$; the Gumbel distribution and the exponential example's normalization are not universal.

Independence is essential to the product calculation. If instead $X_1=X_2=\cdots =X$ for one rate-$\lambda$ exponential random variable, then $M_n=X$ for every $n$. Its distribution function stays $1-e^{-\lambda x}$ for $x\ge 0$ and zero below zero, and its median stays $(\log 2)/\lambda$. Perfect dependence creates no logarithmic growth: repeating the same random variable supplies no new chance of a larger value.

A historical starting point is R. A. Fisher and L. H. C. Tippett's 1928 paper on limiting laws of the largest or smallest sample member, published in volume 24, issue 2, pages 180–190 of the Proceedings of the Cambridge Philosophical Society. Its abstract studies the possible limiting forms and emphasizes that approach to a limit can be slow. ^[ref].

A functional statistic evaluates a statistical functional $T$ at the empirical distribution $F_n=n^{-1}{\sum }_i{\delta }_{X_i}$. The map $T$ acts on distributions, whereas $T(F_n)$ is a statistic. If $T$ has an asymptotic linear representation, thenA square-integrable influence function with mean zero consequently gives asymptotic variance $n^{-1}\int \mathrm{IF}(x;T,F{)}^{2}dF(x)$ by the central limit theorem.

For independent identically distributed mean-zero variables with finite positive variance, the central limit theorem gives a positive eventual lower bound for $\mathbb{P}(S_n/\sqrt{n}\ge K)$ for every finite $K$. The decreasing events ${\bigcup }_{n\ge m}\{S_n/\sqrt{n}\ge K\}$ have probabilities bounded below, so continuity from above of a measure gives positive probability that the limsup is at least $K$. The limsup itself is unchanged by removing any finite initial sum, hence is tail measurable. Kolmogorov zero-one law makes each event “limsup at least $K$” certain; intersect over integer $K$. The result is false for zero variance. One should apply the zero-one law to the limsup, rather than assume the exact threshold infinitely-often event is unaffected by a vanishing change.

For the statistical functional $T(F)=\mu ={\mathbb{E}}_{F}X$, assuming ${\mathbb{E}}_F|X|<\infty$, the contaminated mean is exactlyThus the influence function of the mean is $x-\mu$, and its gross-error sensitivity is infinite on an unbounded sample space. The linear approximation is exact here: any fixed positive contamination fraction allows an arbitrarily large change by moving $x$ far enough. No finite second moment is needed for this derivative, although a finite variance is needed for the usual central limit theorem for the sample mean.

For comparison, the influence function of the sample median iswhen the median $m$ is unique and the probability density function $f$ is positive and continuous at $m$. The sign function records which side of $m$ receives the contaminating mass. The influence function of a quantile supplies the short derivation. With the generalized-inverse quantile convention, contamination exactly at $x=m$ leaves the median at $m$, giving derivative zero there. If $f(m)$ is very small, this bounded influence function can still be large; if $f(m)=0$, the displayed formula is unavailable.

For an independent statistical sample from an Inverse Gaussian distribution with mean $\mu >0$ and known shape $\lambda >0$, the maximum-likelihood estimator of $\mu$ is the sample mean. To prove this, expand the log-likelihood function to get $\ell (\mu )=C-\lambda {\sum }_i{y}_i/(2{\mu }^2)+n\lambda /\mu$. Its score function is $\lambda ({\sum }_i{y}_i-n\mu )/{\mu }^3$, positive below $\bar{y}$ and negative above it. Taking the negative expected second derivative gives one-observation Fisher information $I_1(\mu )=\lambda /{\mu }^3$. Thus the regular asymptotic normality of a maximum likelihood estimator yields $\sqrt{n}(\bar{Y}-\mu )\stackrel{d}{\to }N(0,{\mu }^3/\lambda )$. The same conclusion follows from the central limit theorem and the distribution's variance ${\mu }^3/\lambda$.

For an independent and identically distributed Laplace distribution sample with density $\lambda e^{-\lambda |x-\mu |}/2$ and known rate $\lambda >0$, the sample mean has mean $\mu$ and variance $2/(n{\lambda }^2)$. The central limit theorem gives the displayed asymptotic confidence interval for $\mu$ at confidence level $1-\alpha$. The sample mean need not be a maximum-likelihood estimator for this construction to be valid. The Laplace location and scale maximum likelihood location is a sample median; that is a different choice of statistic.

Laws of large numbers give conditions under which sample means approach a deterministic expected value. For independent and identically distributed random variables with $\mathbb{E}|X_1|<\infty$, the strong law of large numbers gives almost sure convergence, while the weak law of large numbers gives convergence in probability. These conclusions specify different modes of convergence of random variables, rather than a universal numerical accuracy guarantee. The central limit theorem is a separate result about the remaining fluctuations after rescaling.

The logarithm of the stretch after $N$ independent uniform renovations is a sum of independent identically distributed bounded increments. For singular stretching $e^{\gamma }$, $\gamma =\mathrm{arsinh}(|a|/2)$, the increment has Fourier expansion $r+{\sum }_{n\ge 1}(-1{)}^{n+1}{q}^n\cos (2n\theta )/n$, with $q=\tanh \gamma$ and $r=\frac{1}{2}\log (1+a^2/4)$. Orthogonality proves the displayed variance. The central limit theorem gives an approximate central log-normal distribution of ${\lambda }_N$, not an exact description of extreme tails. For $|a|\ll 1$, both the mean and variance of $\log {\lambda }_N$ are $N{a}^2/8$ to leading order. Direct expansion of fixed-power moments gives $⟨{\lambda }_N^p⟩=\exp [Np(p+2)a^2/16+O(N{a}^4)]$; in particular mean stretch grows faster than typical stretch.