Strong law of large numbers (${\overline{X}}_n\stackrel{a.s.}{\to }\mu$)

Let $X_1,X_2,\dots$ be real independent and identically distributed random variables with $\mathbb{E}|X_1|<\infty$, and let $\mu =\mathbb{E}[X_1]$ be their expected value. Their sample mean satisfiesThis is almost sure convergence: with probability one, an entire infinite sequence of observations has averages that eventually stay within every prescribed positive tolerance of $\mu$. The variance need not be finite. The strong law of large numbers implies the weak law of large numbers, which asserts convergence in probability at each increasing sample size.

For independent trials with a Bernoulli distribution of parameter $p$, $X_j$ is one for success and zero for failure. Then ${\overline{X}}_n$ is the observed success proportion, and the strong law of large numbers says ${\overline{X}}_n\to p$ almost surely. This turns a theoretical probability into a quantity one can estimate by repetition. It does not say that each new trial improves the estimate: a long string of failures can move the proportion away from $p$ before later observations bring it back.

The strong law of large numbers gives the limiting location of the sample mean, while the central limit theorem describes its fluctuations when the variance is finite and positive. The two assertions can hold together:The first error shrinks to zero; the second has been enlarged enough to retain a nontrivial probability distribution. The second assertion is convergence in distribution, not a claim that the enlarged error along one sequence approaches a particular normal random variable.

The assumptions make a practical difference. If every observation repeats the same nonconstant Bernoulli random variable, $X_j=X_1$, then ${\overline{X}}_n=X_1$ for all $n$: identical probability distributions alone do not make repeated observations informative. For independent observations with a Standard Cauchy distribution, the Cauchy sample mean has an unchanged distribution, so it does not concentrate around a constant. Its expected value is undefined, and $\mathbb{E}|X_1|=\infty$; neither example contradicts the stated strong law of large numbers.

A brief proof outline in the case of finite variance uses the Kolmogorov maximal inequality for the centered partial sums. The probability that their maximum up to $2^m$ exceeds $\epsilon 2^m$ is at most ${\sigma }^2/({\epsilon }^2{2}^m)$. These bounds are summable, so the Borel-Cantelli lemma controls all sufficiently large such blocks and hence every intermediate sample size. The general case $\mathbb{E}|X_1|<\infty$ requires truncation to control large observations; the argument using finite variance alone does not prove that full version.

For independent identically distributed random variables, the tail-sum formula applied to $|X_1|/\lambda$ shows that ${\sum }_n\mathbb{P}(|X_n|\ge \lambda n)=\infty$ for every $\lambda >0$ when the first absolute moment is infinite. The Borel-Cantelli second lemma, followed by a countable intersection over positive integer $\lambda$, gives the first limsup assertion.

If $|S_n|/n$ had finite limsup along such a sample path, it would eventually be bounded by some finite constant $C$. Then $X_n=S_n-S_{n-1}$ would imply $|X_n|/n\le 2C$ eventually, a contradiction. This is a necessary-integrability counterpart to the strong law of large numbers; without a finite first absolute moment, the normalized sums cannot converge to any finite value.

Take independent random variables equal to $1+n$ and $1-n$ with probability $1/(2n\log n)$ each, and otherwise equal to one. Their first absolute deviations are $1/\log n\to 0$, so $X_n\to 1$ in L1 convergence. The same triangle-inequality argument gives convergence in $L^1$ of their arithmetic averages to one. For $p>1$, however, $\mathbb{E}|X_n-1{|}^p=n^{p-1}/\log n\to \infty$.

The exceptional jumps occur infinitely often almost surely by the Borel-Cantelli second lemma, since $\sum 1/(n\log n)=\infty$. If the arithmetic averages $S_n/n$ converged to any finite value along a sample path, then $X_{n+1}/n=S_n/n-[(n-1)/n][S_{n-1}/(n-1)]$ would tend to zero. At each exceptional jump its magnitude is at least one. Thus the averages have no finite almost sure limit. Independence and convergence of first-moment averages do not alone give a strong law of large numbers.

If the positive factors $U_j$ are independent and identically distributed and $\mathbb{E}|\log U_1|<\infty$, the strong law of large numbers applied to their logarithms gives the displayed almost sure convergence. A negative limit forces the products to tend to zero; a positive limit forces them to tend to infinity. A zero limit alone gives neither conclusion. For a uniform distribution on $(0,b)$, the logarithmic expected value is $\log b-1$, so the products tend to zero almost surely for $b<e$ and to infinity almost surely for $b>e$.

If ${\sup }_n\mathbb{E}|X_n{|}^2<\infty$ and the $X_n$ are pairwise uncorrelated random variables, then $n^{-1}{\sum }_{k=1}^n(X_k-\mathbb{E}{X}_k)\to 0$ almost surely. Independence is unnecessary for this version of the strong law of large numbers. A uniformly bounded martingale difference sequence meets its hypotheses, since the differences have mean zero and are orthogonal in $L^2$.

For independent and identically distributed random variables $X_j$ with $\mathbb{E}|X_1|<\infty$ and $m=\mathbb{E}{X}_1$, the sample mean of $X_j{X}_{j+1}$ tends to $m^2$ almost surely. Split into odd and even indexed products; each subsequence consists of independent and identically distributed random variables that are integrable random variables. Apply the strong law of large numbers separately.