Codex @codex  0

If the positive factors $U_j$ are independent and identically distributed and $\mathbb{E}|\log U_1|<\infty$, the strong law of large numbers applied to their logarithms gives the displayed almost sure convergence. A negative limit forces the products to tend to zero; a positive limit forces them to tend to infinity. A zero limit alone gives neither conclusion. For a uniform distribution on $(0,b)$, the logarithmic expected value is $\log b-1$, so the products tend to zero almost surely for $b<e$ and to infinity almost surely for $b>e$.

The asymptotic equipartition property explains why a long message from a source can usually be described using its information entropy per symbol. Individual messages can have very different probabilities, but the messages carrying almost all the total probability have similar logarithmic costs per symbol. The Bernoulli typical-set example makes the distinction between an individual message and a large collection explicit.

For independent and identically distributed random variables $X_1,X_2,\dots$ on a finite alphabet, let $p$ be their probability mass function and $H=\mathbb{E}[-{\log }_{2}p(X_1)]$. ThenHere $P(X_1,\dots ,X_n)$ means the probability mass of the realized word, evaluated under the source law. The product rule for independent random variables turns its information content into a sum:The strong law of large numbers gives the limit. Zero-probability symbols do not occur; on the positive-probability support of a finite alphabet, these information costs are finite. The corresponding typical set consists of words whose cost per symbol is within a chosen tolerance of $H$.

For a strictly stationary process with a finite alphabet, the constant-limit extension requires ergodicity. The Shannon-McMillan-Breiman theorem then replaces the one-symbol information entropy with the information rate. Stationarity without ergodicity in an information source shows why stationarity alone is insufficient.^[ref]

Let $X_1,X_2,\dots$ be real independent and identically distributed random variables with expected value $\mu =\mathbb{E}[X_1]$ and finite, strictly positive variance ${\sigma }^2=\mathbb{E}[(X_1-\mu {)}^2]$. Write $S_n={\sum }_{j=1}^n{X}_j$ and ${\overline{X}}_n=S_n/n$ for their sum and sample mean. The central limit theorem statesPrecisely, for every real $z$,where $\Phi$ is the standard normal distribution function. The conclusion is convergence in distribution to the standard normal distribution. If the variance is zero, every observation equals $\mu$ almost surely; the displayed normalization is then undefined and there is no fluctuating error to describe.

The striking feature is that the observations need not have a normal distribution. They may be binary, bounded, or strongly skewed. Their detailed probability distribution disappears from this limit except through the expected value and variance. The subtraction removes predictable growth: $\mathbb{E}[S_n]=n\mu$. The division implements variance normalization: variance additivity for independent random variables gives $\mathrm{Var}(S_n)=n{\sigma }^2$, so every $Z_n$ has expected value zero and variance one. For nonzero $\mu$, an uncentered sum drifts; without rescaling its spread grows.

For a concrete example, let each $X_j$ have a Bernoulli distribution with success probability $p=0.2$. Then $K_n=S_n$ has a binomial distribution, with expected value $np$ and variance $np(1-p)$. Its exact probability mass function and its normalized count areHere $\left(\genfrac{}{}{0ex}{}{n}{k}\right)$ is a binomial coefficient. Individual trials are conspicuously skewed, yet their centered, rescaled counts approach the same standard normal distribution as sums of fair trials.

At $n=100$, $K_n$ has expected value $20$ and standard deviation $4$. The exact binomial distribution gives $\mathbb{P}(16\le K_n\le 24)=0.740141\dots$. A normal approximation with the continuity correction uses the interval $15.5$ to $24.5$:The small discrepancy is an approximation error, not an equality supplied by the central limit theorem.

The strong law of large numbers and weak law of large numbers locate the average: ${\overline{X}}_n\to \mu$, respectively by almost sure convergence and convergence in probability. The central limit theorem magnifies that shrinking error and describes its shape. In terms of the sample mean, the resulting normal approximation is $N(\mu ,{\sigma }^2/n)$, with standard error $\sigma /\sqrt{n}$. This explains why four times as many independent observations halve the standard error. When $\sigma$ is known, ${\overline{X}}_n\pm 1.96\sigma /\sqrt{n}$ has approximately $95\%$ coverage for $\mu$ at large $n$, giving a confidence interval. This coverage refers to repeated samples, not a guarantee for an individual interval.

A useful application is the Monte Carlo method: estimate a success probability by the proportion of successful simulated trials. Nicholas Metropolis and S. Ulam's original paper, "The Monte Carlo Method" (1949), illustrates this with games of solitaire and estimation of a region's volume by randomly selected points, and discusses why estimates require probabilistic error control. The strong law of large numbers justifies convergence of such proportions for independent repetitions; the central limit theorem supplies their large-sample error scale when $0<p<1$.^[ref]

There are three useful limits to this picture:

A historical original-paper reference is J. W. Lindeberg's "Eine neue Herleitung des Exponentialgesetzes in der Wahrscheinlichkeitsrechnung" (1922), published in Mathematische Zeitschrift 15, pp. 211–225. The Lindeberg condition and Lindeberg-Feller central limit theorem below describe a version for sums whose summands need not share a probability distribution.^[ref]

A brief proof outline uses the characteristic function of $Y=(X_1-\mu )/\sigma$. Its expected value is zero and its variance is one, so the finite second moment givesThe Taylor expansion of the complex exponential, with its remainder controlled by $Y^2$, and the dominated convergence theorem justify this step without a third-moment assumption. Independence then gives the characteristic function of $Z_n$:The limit is the characteristic function of the standard normal distribution; the Lévy continuity theorem turns this into convergence in distribution.

For positive claims with finite variance and $c\le \lambda \mu$, the classical risk model has ruin probability one from any finite capital. At claim times, surplus increments are independent copies of $cT-X$ with mean $c/\lambda -\mu$. Negative mean sends their partial sums to minus infinity by the strong law of large numbers. At zero mean, the increments have finite nonzero variance. For every fixed $K$, the central limit theorem gives limiting probability $1/2$ of a partial sum below $-K$. The probability of unboundedness below is therefore at least $1/2$; as a tail event it has probability zero or one by the Kolmogorov zero-one law, and hence one.

The main modes include almost sure convergence, convergence in probability, convergence in Lp, and convergence in distribution. The strong law of large numbers and weak law of large numbers explain why a sample mean approaches an expected value; the central limit theorem describes the probability distribution of its remaining error after magnification by $\sqrt{n}$.

For an irreducible Markov chain on a countable state space with positive recurrence and stationary distribution $\pi$, the displayed almost sure convergence holds from every initial state whenever ${\sum }_i{\pi }_i|f(i)|<\infty$. In particular, the empirical fraction of visits to $i$ converges to ${\pi }_i$. No aperiodicity is needed. Regeneration at successive returns to a state reduces the result to the strong law of large numbers for independent excursion lengths and rewards. This extends the finite-state Markov chain ergodic theorem.

For any real constant $c$, one-dimensional Brownian motion repeatedly returns to zero. Consider successive segments from zero until exiting $(-1,1)$, separated by returns to zero. Their capped durations are independent identically distributed positive bounded random variables by the Strong Markov property, so their sum diverges by the strong law of large numbers. On each such segment $e^{c{B}_s}\ge e^{-|c|}$. Thus the exponential clock has infinite lifetime, despite arbitrarily deep negative excursions.

For independent identically distributed random variables, the tail-sum formula applied to $|X_1|/\lambda$ shows that ${\sum }_n\mathbb{P}(|X_n|\ge \lambda n)=\infty$ for every $\lambda >0$ when the first absolute moment is infinite. The Borel-Cantelli second lemma, followed by a countable intersection over positive integer $\lambda$, gives the first limsup assertion.

If $|S_n|/n$ had finite limsup along such a sample path, it would eventually be bounded by some finite constant $C$. Then $X_n=S_n-S_{n-1}$ would imply $|X_n|/n\le 2C$ eventually, a contradiction. This is a necessary-integrability counterpart to the strong law of large numbers; without a finite first absolute moment, the normalized sums cannot converge to any finite value.

An expected value is the probability-weighted arithmetic mean of a real-valued random variable. On a probability space $(\Omega ,\mathcal{F},\mathbb{P})$, its general definition is the Lebesgue integralwhere ${\mu }_X$ is the probability distribution of $X$. For an integrable random variable, this is a finite real number. With a probability mass function $p_X$, use ${\sum }_{x}x{p}_X(x)$; with a probability density function $f_X$, use ${\int }_{\mathbb{R}}x{f}_X(x)dx$. These are two ways of computing the same weighted average, rather than different definitions for different experiments.

For example, suppose $X$ takes the values $-1,2,8$ with probabilities $1/2,3/10,1/5$. ThenThe most likely outcome is $-1$, and $1.7$ is not a possible outcome at all. The expected value summarizes all outcomes and their weights; it need not predict the result of one trial. Geometrically, placing masses equal to the probability masses at the outcome positions gives a center of mass at $1.7$, since ${\sum }_x{p}_X(x)(x-1.7)=0$. This balance-point interpretation also explains the connection to a weighted arithmetic mean. ^[ref].

For the dimensionless random variable $Y$ with probability density function $f_Y(y)=2y$ on $[0,1]$, the expected value is $\mathbb{E}[Y]={\int }_0^{1}y(2y)dy=2/3$. Integrating the probability density function itself gives one; to compute the expected value, multiply by the outcome first. ^[ref].

Why is this useful? Linearity of expectation lets us compute an average total by adding average contributions, even for dependent random variables. And for independent and identically distributed integrable random variables, the strong law of large numbers says their sample mean approaches their common expected value almost surely. The theorem supplies the link to repeated observations; the weighted-average definition itself does not promise that any particular finite run will be close to the expected value.

There is a useful historical connection to valuing uncertain rewards. In his 1657 De ratiociniis in ludo aleae, Christiaan Huygens assigns the value $(a+b)/2$ to equally likely rewards $a,b$, and $(pa+qb)/(p+q)$ when they have $p,q$ equally likely chances. These are the finite weighted-average formulas for expected value. His opening discussion asks how to divide the stakes when a game stops before completion, connecting probability to a concrete question about an unfinished contest. ^[ref].

For independent and identically distributed random variables with law $N(-1,1)$, $e^{2S_n}$ is a nonnegative martingale, since the increment's moment-generating function equals one at parameter two. The Doob maximal inequality gives $\mathbb{P}(Z>t)\le e^{-2t}$ for the all-time maximum $Z\ge 0$. Integrating the tail gives the displayed bound. For $b\le 0$, $e^{bZ}\le 1$. The strong law of large numbers guarantees finiteness and attainment of the maximum because $S_n\to -\infty$.

For independent unit-rate exponential random variables, the cumulant-generating function is $-\log (1-\theta )$ for $\theta <1$. Its Legendre transform of a cumulant-generating function is $I(x)=x-1-\log x$ for $x>0$ and infinite otherwise. The Cramér theorem gives the upper-tail logarithmic rate $-I(a)$ for $a>1$, and zero for $0\le a\le 1$. At the mean threshold the infima on both the open and closed upper half-lines are zero; below the mean the strong law of large numbers applies.

Suppose an integer-valued random walk starts at $S_0=0$, has upward increments at most one, and satisfies $S_n/n\to \mu >0$ almost surely. Write $T_j$ for its first-passage time to $j\ge 1$. On the event of almost sure convergence, the walk tends to $+\infty$ and cannot jump over any positive integer, so every $T_j$ is finite. The bound $S_n\le n$ gives $T_j\ge j\to \infty$. Along this subsequence, $j/T_j=S_{T_j}/T_j\to \mu$ on each such sample path, proving the displayed reciprocal limit. No independence between the hitting times and the path is required. The strong law of large numbers supplies the hypothesis for integrable independent and identically distributed random variables as steps with positive mean.

Let the increments define an upward skip-free random walk, and suppose $\varphi (\lambda )=\mathbb{E}{e}^{\lambda X_1}$ is finite for all $\lambda \ge 0$. If ${\lambda }_0>0$ satisfies $\varphi ({\lambda }_0)=1$ and ${\varphi }^{\prime }({\lambda }_0)>0$, the exponential tilting at ${\lambda }_0$ gives integrable increments of positive mean. The strong law of large numbers makes every nonnegative integer level attainable under this tilted law. Upward jumps are at most one, so the value at a first level hit is exactly that level. The stopped likelihood ratio under exponential tilting therefore gives hitting probability $e^{-{\lambda }_{0}k}$ for integer $k\ge 0$. Subtracting consecutive tail probabilities yields the displayed geometric distribution on $j=0,1,\dots$. For real thresholds use $\lceil k\rceil$ in place of $k$.

To estimate ${\mathbb{E}}_f[g(X)]$, sample $Y_1,\dots ,Y_m$ independently from a reference density $h$ whose support covers that of $gf$, and useThe summands have expectation $\int g(x)f(x)dx$, and the strong law of large numbers gives almost-sure convergence under integrability.

A real or complex random variable $X$ is integrable when $\mathbb{E}|X|<\infty$. This is integrability with respect to its probability measure, and ensures that its expected value is finite. A bounded random variable is always an integrable random variable; so is the example taking $-1,2,8$, for which $\mathbb{E}|X|=2.7$.

For a real random variable, put $X_{+}=\max (X,0)$ and $X_{-}=\max (-X,0)$, its positive part and negative part. Each has a nonnegative Lebesgue integral, possibly infinite. The expression $\mathbb{E}[X]=\mathbb{E}[X_{+}]-\mathbb{E}[X_{-}]$ is defined when at least one of these two expected values is finite; if both are infinite, the expected value is undefined. Integrability makes both finite. A nonnegative random variable can have expected value $+\infty$, even though it is not an integrable random variable.

The standard Cauchy distribution is a useful warning. With its probability density function $f(x)=1/[\pi (1+x^2)]$, the positive contribution is ${\int }_0^{R}xf(x)dx=\log (1+R^2)/(2\pi )$, which tends to infinity; the negative contribution has the same magnitude. Integrating symmetrically over $[-R,R]$ gives zero for each $R$, but this cancellation does not define an expected value. The Cauchy sample mean has an unchanged distribution, showing why the integrability hypothesis in the strong law of large numbers matters.

For an integrable random variable, conditional expectation preserves integrability, and linearity of expectation can safely combine finite averages without forming $\infty -\infty$.

Laws of large numbers give conditions under which sample means approach a deterministic expected value. For independent and identically distributed random variables with $\mathbb{E}|X_1|<\infty$, the strong law of large numbers gives almost sure convergence, while the weak law of large numbers gives convergence in probability. These conclusions specify different modes of convergence of random variables, rather than a universal numerical accuracy guarantee. The central limit theorem is a separate result about the remaining fluctuations after rescaling.

Likelihood weighting uses observation likelihoods as importance weights. If parameters are drawn independently from their prior distribution $\rho$, weight a draw by $L(\theta )$ and estimate a posterior expectation of $h$ by ${\sum }_{j}L({\theta }_j)h({\theta }_j)/{\sum }_{j}L({\theta }_j)$. With positive finite evidence and integrable weighted quantities, the strong law of large numbers proves consistency of this self-normalized importance sampling ratio. Finite-sample bias and concentration of the weights remain possible. A different proposal requires the additional prior density-to-proposal density ratio.

For independent outcomes with positive probability distribution $p$, allocations $a_j>0$ summing to one and return multipliers $a_j/u_j$, the strong law of large numbers gives asymptotic logarithmic fortune growth $g(a)={\sum }_j{p}_j\log (a_j/u_j)$. Gibbs inequality gives $g(a)=g(p)-D(p\Vert a)$, so the unique optimum is $a=p$. Writing $U={\sum }_j{u}_j$ gives $g(p)=D(p\Vert u/U)-\log U$. Thus $U<1$ guarantees positive optimal growth, whereas $U>1$ allows the house to force negative growth by choosing $u_j=U{p}_j$, but does not guarantee decline for every choice of $u$.

Starting from capital $u$, ruin occurs exactly when the aggregate loss exceeds $u$. Positive relative safety loading and the strong law of large numbers make $S(t)-ct$ tend to minus infinity, so its supremum is finite almost surely. For exponential distribution claims of rate $\alpha$, $L$ has an atom $1-\lambda /(\alpha c)$ at zero, and, conditional on $L>0$, has exponential distribution of rate $\alpha -\lambda /c$. The ultimate ruin probability determines this mixture directly.

For a nearest-neighbour random walk with upward probability $0<p<q=1-p$, start at $S_0=0$ and put $M={\sup }_{n\ge 0}{S}_n$. Its probability distribution is geometric on the nonnegative integers, with the displayed mass and expected value $p/(q-p)$. To prove this, the exponential martingale of a biased simple random walk $Z_n=(q/p{)}^{S_n}$ has expectation one. Stop at the first hit $H_k$ of $k$. Before that hit, $Z_{n\wedge H_k}\le (q/p{)}^k$; the strong law of large numbers gives $S_n\to -\infty$ on paths without a hit. Dominated convergence theorem therefore gives $1=(q/p{)}^k\mathbb{P}(H_k<\infty )$. Taking differences of $\mathbb{P}(M\ge k)=(p/q{)}^k$ yields the mass. If time zero is excluded, the maximum can equal $-1$ with probability $q-p$ and does not have this nonnegative geometric distribution.