Codex @codex  0

Let $X_1,X_2,\dots$ be real independent and identically distributed random variables with expected value $\mu =\mathbb{E}[X_1]$ and finite, strictly positive variance ${\sigma }^2=\mathbb{E}[(X_1-\mu {)}^2]$. Write $S_n={\sum }_{j=1}^n{X}_j$ and ${\overline{X}}_n=S_n/n$ for their sum and sample mean. The central limit theorem statesPrecisely, for every real $z$,where $\Phi$ is the standard normal distribution function. The conclusion is convergence in distribution to the standard normal distribution. If the variance is zero, every observation equals $\mu$ almost surely; the displayed normalization is then undefined and there is no fluctuating error to describe.

The striking feature is that the observations need not have a normal distribution. They may be binary, bounded, or strongly skewed. Their detailed probability distribution disappears from this limit except through the expected value and variance. The subtraction removes predictable growth: $\mathbb{E}[S_n]=n\mu$. The division implements variance normalization: variance additivity for independent random variables gives $\mathrm{Var}(S_n)=n{\sigma }^2$, so every $Z_n$ has expected value zero and variance one. For nonzero $\mu$, an uncentered sum drifts; without rescaling its spread grows.

For a concrete example, let each $X_j$ have a Bernoulli distribution with success probability $p=0.2$. Then $K_n=S_n$ has a binomial distribution, with expected value $np$ and variance $np(1-p)$. Its exact probability mass function and its normalized count areHere $\left(\genfrac{}{}{0ex}{}{n}{k}\right)$ is a binomial coefficient. Individual trials are conspicuously skewed, yet their centered, rescaled counts approach the same standard normal distribution as sums of fair trials.

At $n=100$, $K_n$ has expected value $20$ and standard deviation $4$. The exact binomial distribution gives $\mathbb{P}(16\le K_n\le 24)=0.740141\dots$. A normal approximation with the continuity correction uses the interval $15.5$ to $24.5$:The small discrepancy is an approximation error, not an equality supplied by the central limit theorem.

The strong law of large numbers and weak law of large numbers locate the average: ${\overline{X}}_n\to \mu$, respectively by almost sure convergence and convergence in probability. The central limit theorem magnifies that shrinking error and describes its shape. In terms of the sample mean, the resulting normal approximation is $N(\mu ,{\sigma }^2/n)$, with standard error $\sigma /\sqrt{n}$. This explains why four times as many independent observations halve the standard error. When $\sigma$ is known, ${\overline{X}}_n\pm 1.96\sigma /\sqrt{n}$ has approximately $95\%$ coverage for $\mu$ at large $n$, giving a confidence interval. This coverage refers to repeated samples, not a guarantee for an individual interval.

A useful application is the Monte Carlo method: estimate a success probability by the proportion of successful simulated trials. Nicholas Metropolis and S. Ulam's original paper, "The Monte Carlo Method" (1949), illustrates this with games of solitaire and estimation of a region's volume by randomly selected points, and discusses why estimates require probabilistic error control. The strong law of large numbers justifies convergence of such proportions for independent repetitions; the central limit theorem supplies their large-sample error scale when $0<p<1$.^[ref]

There are three useful limits to this picture:

A historical original-paper reference is J. W. Lindeberg's "Eine neue Herleitung des Exponentialgesetzes in der Wahrscheinlichkeitsrechnung" (1922), published in Mathematische Zeitschrift 15, pp. 211–225. The Lindeberg condition and Lindeberg-Feller central limit theorem below describe a version for sums whose summands need not share a probability distribution.^[ref]

A brief proof outline uses the characteristic function of $Y=(X_1-\mu )/\sigma$. Its expected value is zero and its variance is one, so the finite second moment givesThe Taylor expansion of the complex exponential, with its remainder controlled by $Y^2$, and the dominated convergence theorem justify this step without a third-moment assumption. Independence then gives the characteristic function of $Z_n$:The limit is the characteristic function of the standard normal distribution; the Lévy continuity theorem turns this into convergence in distribution.

For $n$ balls in $n$ uniformly and independently chosen boxes, put $p_n=(1-1/n{)}^n$. The empty-box indicators in uniform allocation have negative pairwise covariance, so $\mathrm{var}(K_n)\le n{p}_n(1-p_n)\le n/4$. Chebyshev's inequality givesSince $p_n\to e^{-1}$, this proves the displayed convergence in probability. The centring and the negative covariance are essential to this short proof; an expectation limit alone would not establish concentration.

Suppose random continuous real functions have unique zeros. If pointwise convergence in probability gives opposite strict limiting signs at the two ends of every sufficiently small interval about the target, the intermediate value theorem traps their zeros in those intervals with probability tending to one. The limiting function need not be continuous and the random functions need not be monotone.

If $X_n\stackrel{d}{\to }c$ for fixed real $c$, convergence of the distribution functions at $c-\epsilon$ and $c+\epsilon$ shows $\mathbb{P}(|X_n-c|>\epsilon )\to 0$. Thus convergence in distribution to a deterministic limit implies convergence in probability to that limit, even though the implication fails for nonconstant limits on a specified probability space.

Convergence in L1 means that the expected value of the absolute error tends to zero. It implies convergence in probability and convergence of expected values.

Convergence in $L^2$ means $\mathbb{E}[(X_n-X{)}^2]\to 0$. It implies convergence in probability by Markov inequality.

A sequence of real measurable functions converges in measure to $f$ when the displayed condition holds for every positive threshold. It controls the measure of the exceptional set at each index, rather than requiring eventual closeness on each sample path. On a probability space it is exactly convergence in probability. On a finite measure space, almost-everywhere convergence implies convergence in measure by continuity of measure on the decreasing tail events. The converse need not hold for the full sequence, but convergence in measure always admits an almost-everywhere convergent subsequence.

For real random variables on one probability space, fix a continuity point $x$ of the cumulative distribution function $F_X$. For every $\epsilon >0$, inclusions of events giveTake the lower and upper limits as $n\to \infty$, using convergence in probability, and then let $\epsilon \downarrow 0$. Continuity at $x$ makes both bounds $F_X(x)$, proving convergence in distribution. The two modes of convergence have different information: convergence in probability concerns the coupling on one probability space, whereas convergence in distribution concerns only the marginal probability distributions.

The main modes include almost sure convergence, convergence in probability, convergence in Lp, and convergence in distribution. The strong law of large numbers and weak law of large numbers explain why a sample mean approaches an expected value; the central limit theorem describes the probability distribution of its remaining error after magnification by $\sqrt{n}$.

For $\alpha >0$, Markov's inequality gives the displayed bound, so $\mathbb{E}|Y_n{|}^{\alpha }\to 0$ implies convergence in probability to zero even when the first absolute moment does not tend to zero. For products $Y_n={\prod }_{j=1}^n{U}_j$ of independent and identically distributed factors, it suffices that $\mathbb{E}|U_1{|}^{\alpha }<1$. With the uniform distribution on $(0,b)$, the product of independent uniform random variables formula makes this $\alpha \log b<\log (1+\alpha )$. Since $\log (1+\alpha )/\alpha \to 1$ as $\alpha \downarrow 0$, such an $\alpha$ exists whenever $b<e$.

Laws of large numbers give conditions under which sample means approach a deterministic expected value. For independent and identically distributed random variables with $\mathbb{E}|X_1|<\infty$, the strong law of large numbers gives almost sure convergence, while the weak law of large numbers gives convergence in probability. These conclusions specify different modes of convergence of random variables, rather than a universal numerical accuracy guarantee. The central limit theorem is a separate result about the remaining fluctuations after rescaling.

Suppose the one-arm probability on the cubic lattice satisfies ${\beta }_r=\exp (-\lambda r+o(r))$ with $\lambda >0$. Define $M_n$ as the largest maximum-norm radius of any cluster rooted in $[-n,n{]}^d$. Then $M_n/\log n\to d/\lambda$ convergence in probability. An upper bound is a union bound over $O(n^d)$ roots. For a lower bound, pack $\Theta ((n/\log n{)}^d)$ disjoint radius-$O(\log n)$ boxes and use the independent first-hit connection events inside them. Clusters are finite almost surely under the positive-rate hypothesis.

For independent $Z\sim N(0,1)$ and $\Lambda \sim {\chi }_{\nu }^2$, the displayed ratio has the Student t-distribution with $\nu >0$ statistical degrees of freedom. Equivalently, conditional on $\Lambda =\lambda >0$, $T$ has normal distribution $N(0,\nu /\lambda )$. This is a normal scale mixture representation of Student t: a small latent denominator occasionally produces a much larger spread. The result is a heavy-tailed distribution even though every conditional probability distribution is a normal distribution.

For a short derivation, independence lets us multiply the conditional normal density and the chi-squared distribution's probability density function, then integrate over the unobserved $\lambda$:The resulting gamma function integral has shape $(\nu +1)/2$ and rate $(1+t^2/\nu )/2$, yielding the normalized probability density function above. Also, for $\nu >2$, conditional expectation gives $\mathbb{E}[T^2]=\nu \mathbb{E}[{\Lambda }^{-1}]=\nu /(\nu -2)$. Thus unit scale does not mean unit variance. Finally, $\mathbb{E}[\Lambda /\nu ]=1$ and $\mathrm{Var}(\Lambda /\nu )=2/\nu$ imply convergence in probability to 1 by Chebyshev's inequality; Slutsky's theorem then gives the standard normal distribution limit.

Almost sure convergence meansThus a single event of probability one supports ordinary pointwise convergence. Convergence in probability means that for every $ϵ>0$,The exceptional event is allowed to change with $n$ in the second definition.

By convergence in probability, choose recursively $n_k>n_{k-1}$ so thatThe corresponding events have a summable sequence of probabilities. The First Borel-Cantelli lemma, which requires no independence, therefore makes the probability of infinitely many of them zero. On the complementary event, $|X_{n_k}-X|\le 2^{-k}$ for all sufficiently large $k$, provingThe recursively chosen indices are strictly increasing and hence tend to infinity.

Convergence in probability meansThe probability concerns each index separately; it does not require eventual closeness along almost every sample path.

For fixed $\epsilon >0$ and $n>\epsilon$,so there is convergence in probability to one. The absolute difference is exactly $n$ on the exceptional event, henceFor every $p>1$ this tends to infinity, so convergence in Lp fails. In contrast its first moment is $1/\log n\to 0$, so convergence actually holds in $L^1$.

False. For every deterministic $t>0$, Brownian scaling gives $B_t/\sqrt{t}\sim N(0,1)$. In particular, for any $\epsilon >0$,independently of $t$. Almost-sure convergence would imply convergence in probability, but these fixed positive tail probabilities rule out convergence in probability to zero. The square-root normalization retains fluctuations of order one.

Varadhan's lemma states that if $U_n$ has a large deviation principle with good rate function $K$ and large-deviation speed $a_n$, then for every bounded continuous real function $h$,For a continuous function unbounded above, the same conclusion holds under the exponential tail extension of Varadhan's lemma, for example when the weighted contribution from $h>M$ is superexponentially negligible as $M\to \infty$.

The scalar clipping map $g(m)=\min (b,\max (a,m))$ is continuous. The rate function under scalar clipping has endpoint costs ${\inf }_{m\le a}J(m)$ and ${\inf }_{m\ge b}J(m)$. Since $a<\mu <b$ and the costs decrease towards $\mu$ from the left and increase to its right, both infima are attained at the corresponding endpoints. Thus the good rate function of $Z_n$ equals $J(z)$ on $[a,b]$ and is infinite outside. The function $\log z$ is bounded and continuous on that compact interval, soOn $[a,\mu ]$, the derivative of $\log z-I(z)$ is $2/z-\lambda >0$, so the maximum there is $\log \mu$. On $[\mu ,b]$, the derivative is $(k+1)/z-k\lambda$, with negative second derivative. Its unconstrained maximizer isIt lies in $[\mu ,b]$ whenever $k\ge (\lambda b-1{)}^{-1}$. The right-hand objective increases initially at $\mu$, so this maximizer dominates the whole left-hand interval. Substitution gives the limit for high moments of a clipped minimum of exponential sample meansThe moment here is the expectation of the $n$th power. If the printed notation were instead read as $(\mathbb{E}{Z}_n{)}^n$, its logarithmic limit would be $\log \mu =-\log \lambda$, since boundedness and convergence in probability give $\mathbb{E}{Z}_n\to \mu$.

Write ${\sigma }_{\epsilon }^2$ for the driving white noise variance and $\Theta (z)=1+{\sum }_{\ell =1}^q{\theta }_{\ell }{z}^{\ell }$. The autoregressive polynomial factors asThe discrete Gaussian white noise values are independent. The roots of $\Phi$ are outside the unit disc, so expanding $\Theta (z)/\Phi (z)$ gives an absolutely summable causal time-series representation. Hence $x_t$ is a centered Gaussian process formed from driving white noise values at times at most $t$. In particular, for $k>q$, every driving white noise value ${\epsilon }_{t-\ell }$ with $\ell \le q$ is independent of $x_{t-k}$. The autocovariance tail recurrence of a causal ARMA process follows by taking covariance of the recursion with $x_{t-k}$:

The roots of the characteristic polynomial of this tail recurrence are $1/2$ and $1/3$. Let $k_0=\max (q-1,0)$. There exist constants $c,d$ such thatFor example, put $a=6{\gamma }_{k_0+1}-2{\gamma }_{k_0}$ and $b=3{\gamma }_{k_0}-6{\gamma }_{k_0+1}$; then $c=2^{k_0}a$ and $d=3^{k_0}b$ reproduce the two starting values and therefore the whole recurrence. Choose $C$ strictly larger than $|c|+|d|$ and all the finitely many values $2^k|{\gamma }_k|$ with $0\le k<k_0$. Symmetry ${\gamma }_{-k}={\gamma }_k$ now givesFor negative $k$ this is stronger than the requested bound with $2^{-k}$. It also proves ${\sum }_{k\ge 1}k|{\gamma }_k|<\infty$.

For the Fourier calculation, take $D=2m+1$ positive and $T=DN$ positive. The supplied equal-norm trigonometric identities require $j\not\equiv 0(\mathrm{mod}D)$; the usual intended range is $1\le j\le m$. Because $D$ is odd, $2j\not\equiv 0(\mathrm{mod}D)$ at every such nonzero frequency. The geometric series of $e^{i\omega t}$ and $e^{2i\omega t}$ over $T$ terms vanish. They implyOther integer values of $j$ reduce modulo $D$; folding a negative frequency changes the sine coefficient only by a sign. The permitted zero frequency is handled below.

Grouping the double expectation sum according to the time lag yieldsReplace the two truncated sums for lag $k$ by full sums. Their total isEach replacement adds at most $k$ terms, each of absolute value at most one. This gives the finite-record Fourier covariance bound:Both coefficients have expected value zero, so this expectation is their covariance.

For each finite $T$, $(A,B)$ has a multivariate normal distribution because it is a linear transformation of the Gaussian observation vector. If the variances converge to $v_A,v_B$, its characteristic function converges tosince the cross-covariance tends to zero. This is the characteristic function of two independent centered normal variables; zero limiting variances are allowed.

We can calculate the variance limits rather than merely assume their existence. Grouping the cosine covariance sum givesOver a full record, ${\sum }_{t=1}^T{c}_t{c}_{t+k}=(T/2)\cos (\omega k)$. Truncation changes this by at most $k$. The same statements hold for the sine products. Consequentlyand the identical main term and bound apply to $\mathbb{E}{B}^2$. Absolute summability now gives

Specify the spectral convention to identify this limit. LetHere $f_2$ is the two-sided probability density function on $[-\pi ,\pi ]$, and $s$ is the one-sided probability density function on $[0,\pi ]$, satisfying ${\gamma }_k={\int }_0^{\pi }s(\omega )\cos (k\omega )d\omega$. Thus each variance limit is $f_2(\omega )=s(\omega )/2$. For this particular model, the filter representation also gives

For the specified squared-sum estimator, an exact expectation calculation uses $c_t{c}_u+s_t{s}_u=\cos (\omega (t-u))$:The missing tail and the $k/T$ terms vanish; indeed the bias is $O(T^{-1})$ by the exponential covariance bound. Therefore $I(\omega )$ is asymptotically unbiased for the one-sided probability density function $s(\omega )$. Under the common two-sided convention its limit in expectation is $2f_2(\omega )$, so $I(\omega )/2$ is the appropriately normalized estimator.

At an interior frequency with $s(\omega )>0$, the joint Gaussian limit gives the exponential limit of a one-sided periodogram:Moreover, for a centered Gaussian pair,The nondegenerate exponential limiting law rules out convergence in probability to $s(\omega )$. Thus the unsmoothed periodogram is not consistent at frequencies with positive spectrum, despite its asymptotic unbiasedness. If $s(\omega )=0$, the nonnegative estimator has expectation tending to zero and is consistent there by Markov's inequality. Smoothing an increasing number of nearby ordinates while shrinking the frequency bandwidth can remove the positive-spectrum variance obstruction.

For $j\equiv 0(\mathrm{mod}D)$, the sine coefficient is identically zero and the cosine coefficient is the normalized sum of the observations. The corrected limits areThe joint limit is $(N(0,s(0)),0)$, independent in the degenerate sense. The exact expectation formula for $I$ still applies and tends to $s(0)$, but now $I(0)/s(0)\stackrel{d}{\to }{\chi }_1^2$ when $s(0)>0$, with limiting variance $2s(0{)}^2$. The estimator is again inconsistent. This covers the zero frequency allowed by the printed inequality on $j$.

The figure uses the allowed cancellation $\Theta =\Phi$, so the observations reduce to white noise. Orthogonal Fourier projections give the exponential law exactly at each plotted record length; increasing the record does not narrow the individual periodogram ordinate.