Convergence in probability

Random variables $X_n$ converge in probability to $X$ whenfor every $\epsilon >0$.

For real random variables on one probability space, fix a continuity point $x$ of the cumulative distribution function $F_X$. For every $\epsilon >0$, inclusions of events giveTake the lower and upper limits as $n\to \infty$, using convergence in probability, and then let $\epsilon \downarrow 0$. Continuity at $x$ makes both bounds $F_X(x)$, proving convergence in distribution. The two modes of convergence have different information: convergence in probability concerns the coupling on one probability space, whereas convergence in distribution concerns only the marginal probability distributions.

For $\alpha >0$, Markov's inequality gives the displayed bound, so $\mathbb{E}|Y_n{|}^{\alpha }\to 0$ implies convergence in probability to zero even when the first absolute moment does not tend to zero. For products $Y_n={\prod }_{j=1}^n{U}_j$ of independent and identically distributed factors, it suffices that $\mathbb{E}|U_1{|}^{\alpha }<1$. With the uniform distribution on $(0,b)$, the product of independent uniform random variables formula makes this $\alpha \log b<\log (1+\alpha )$. Since $\log (1+\alpha )/\alpha \to 1$ as $\alpha \downarrow 0$, such an $\alpha$ exists whenever $b<e$.

Uniform convergence in probability means that the supremum of the absolute discrepancy on the entire index set tends to zero in probability. It controls evaluation at random indices in that set, unlike pointwise convergence in probability. On a fixed compact index set it is a special case of uniform convergence on compacts in probability.

A sequence of random variables $(X_n)$ is bounded in probability when ${\lim }_{K\to \infty }{\sup }_n\mathbb{P}(|X_n|>K)=0$. A sequence converging in probability to a finite random variable has this property, and its product with a sequence tending to zero in probability also tends to zero in probability.

The notation $Z_n=O_{\mathbb{P}}(a_n)$ means that $Z_n/a_n$ is bounded in probability asymptotically: for every $\epsilon >0$ some $C$ gives ${\mathrm{limsup}}_n\mathbb{P}(|Z_n|>C{a}_n)\le \epsilon$. A tail bound $\mathbb{P}(|Z_n|>z{a}_n)\le C_0/z^2$ is sufficient. In statements involving bandwidths, specify whether the constants must be uniform over bandwidth sequences.

A bound on ${\sup }_x\mathbb{P}(|Z_n(x)|>t)$ does not give the same bound on $\mathbb{P}({\sup }_x|Z_n(x)|>t)$. Even independent standard normal random variables each have uniformly bounded tails while their Gaussian maximum over $N$ indices grows at scale $\sqrt{\log N}$. This distinction matters in nonparametric statistics when a shrinking bandwidth permits many spatial windows.

If $X_n\stackrel{d}{\to }c$ for fixed real $c$, convergence of the distribution functions at $c-\epsilon$ and $c+\epsilon$ shows $\mathbb{P}(|X_n-c|>\epsilon )\to 0$. Thus convergence in distribution to a deterministic limit implies convergence in probability to that limit, even though the implication fails for nonconstant limits on a specified probability space.

Let $X_1,X_2,\dots$ be real independent and identically distributed random variables with $\mathbb{E}|X_1|<\infty$ and expected value $\mu =\mathbb{E}[X_1]$. The weak law of large numbers states that their sample mean has convergence in probability to $\mu$: for every fixed $\epsilon >0$,Thus a large sample is unlikely to give an average with a specified fixed-size error. The strong law of large numbers makes the stronger statement of almost sure convergence along an entire sequence. Under the displayed assumptions both laws apply; their names distinguish the conclusions, not two competing predictions.

When the variance ${\sigma }^2$ is finite, variance additivity for independent random variables gives $\mathrm{Var}({\overline{X}}_n)={\sigma }^2/n$. The Chebyshev inequality immediately supplies a useful quantitative form and a brief proof:For independent trials with a Bernoulli distribution of parameter $p=0.2$, the expected value is $0.2$ and the variance is $0.16$. At $n=1000$, the chance of an error of at least $0.1$ is therefore at most $0.016$. The exact binomial distribution gives a much smaller probability, as the figure under the strong law of large numbers shows. The bound is conservative, but it is valid for every sample size under these assumptions and does not require a normal approximation.

The weak law of large numbers addresses a fixed tolerance $\epsilon$. The central limit theorem instead resolves errors on the shrinking scale $\sigma /\sqrt{n}$ and gives their approximate normal distribution. In particular, the weak law of large numbers alone supplies neither a bell-shaped error law nor a universal rate of approach. For example, a Pareto distribution with minimum one and shape $3/2$ has expected value three but infinite variance: both the weak law of large numbers and strong law of large numbers apply, while the finite-variance statement of the central limit theorem does not.

Suppose the variances of a sequence of random variables are bounded by $C$ and $\mathrm{Cov}(X_i,X_j)=0$ whenever $|i-j|>r$, for a fixed nonnegative integer $r$. The Cauchy-Schwarz inequality bounds each remaining covariance in absolute value by $C$. In the variance of a sum, there are at most $rn$ off-diagonal pairs, proving the displayed bound. The Chebyshev inequality makes the average minus its mean converge to zero in probability. If all means equal $\mu$, the average therefore converges in probability to $\mu$. Independence of every summand is unnecessary.

Let $X$ have a symmetric probability distribution with $\mathbb{P}(|X|>x)=1/(x\log x)$ for $x\ge e$, mass $1-1/e$ at zero, and no mass with $0<|X|<e$. Then $\mathbb{E}|X|=\infty$, but independent copies satisfy the weak law of large numbers with limit zero. Indeed $n\mathbb{P}(|X|>n)=1/\log n\to 0$. Their symmetric truncations $Y_{n,j}=X_j{1}_{\{|X_j|\le n\}}$ have mean zero and second moment $O(n/\log n)$, so Chebyshev's inequality gives $n^{-1}{\sum }_{j\le n}{Y}_{n,j}\to 0$ in probability. A union bound shows that the original and truncated sums differ with probability tending to zero. This also gives a differentiable characteristic function at zero whose derivative cannot be interpreted as an absolutely convergent expectation.

For independent and identically distributed random variables with characteristic function $\varphi$, suppose their sample mean converges in probability to a finite constant $a$. Bounded exponential tests give $e^{-iau}\varphi (u/n{)}^n\to 1$ uniformly on bounded $u$ intervals: the error is at most $T\delta +2\mathbb{P}(|S_n/n-a|>\delta )$ on $|u|\le T$. Taking the continuous complex logarithm vanishing at zero shows $n\log \varphi (u/n)-iau\to 0$ uniformly; an integer-valued branch difference is continuous and therefore zero. For arbitrary $h\to 0$, put $n=\lfloor 1/|h|\rfloor$ and $u=nh$, so $1/2\le |u|\le 1$. Dividing the uniform limit by $u$ proves $\log \varphi (h)/h\to ia$, and thus ${\varphi }^{\prime }(0)=ia$. No absolute first moment is assumed.

A class $\mathcal{G}$ satisfies a uniform law of large numbers when its empirical means converge uniformly to their expectations:

A measurable-function class is pointwise separable if it has a countable subclass such that every member is the pointwise limit of a sequence from that subclass. With an integrable common envelope, dominated convergence theorem makes both empirical averages and population expectations converge along each such sequence. Thus the supremum of the absolute empirical errors equals the supremum over the countable subclass and is measurable. A finite integrable bracketing cover provides an integrable envelope by taking the maximum absolute value of all its endpoints.

A bracket $[\ell ,u]$ contains every measurable function $h$ satisfying $\ell \le h\le u$ pointwise. For a probability measure $P$, its $L^1(P)$ width is $P(u-\ell )$, with integrable endpoints. Finite bracketing at every positive width is a sufficient complexity condition for a uniform law of large numbers. Pointwise inequalities, or inequalities outside a single common null set, allow empirical inequalities to hold simultaneously throughout the class.

A measurable function $M$ is an integrable envelope of a function class for a class $\mathcal{H}$ under a probability law $P$ if $|h(x)|\le M(x)$ for every $h\in \mathcal{H}$ and $PM<\infty$. Such an envelope permits the dominated convergence theorem to control shrinking function brackets.

Suppose an integrable measurable-function class can be covered by finitely many function brackets of every positive $L^1(P)$ width. For an independent sample with common law $P$, the empirical measure satisfies ${\sup }_h|P_{n}h-Ph|\to 0$ on a common probability-one event. For one finite $\epsilon$-cover, the supremum is bounded by $\epsilon$ plus the largest empirical error among its endpoints. The strong law of large numbers makes that finite maximum vanish. Taking a countable sequence of widths decreasing to zero proves the assertion. A measurable supremum can be obtained from a pointwise separable function class; otherwise the probability-one-event formulation expresses the same pathwise conclusion.

If $\Theta$ is compact, $q(\theta ,x)$ is continuous in $\theta$ and measurable in $x$, and $\mathbb{E}{\sup }_{\Theta }|q(\theta ,X)|<\infty$, then an independent and identically distributed sample satisfies ${\sup }_{\Theta }|P_n{q}_{\theta }-P{q}_{\theta }|\to 0$ almost surely. Uniform continuity for each $x$, the dominated convergence theorem, and finite parameter nets produce arbitrarily narrow function brackets.

The empirical distribution function converges uniformly almost surely to the population distribution function.

For a cumulative distribution function $G$, let $Q(u)=\inf \{x:G(x)\ge u\}$, $0<u<1$. Right continuity gives $Q(u)\le x$ exactly when $u\le G(x)$, including at atoms. Thus applying $Q$ to independent uniform variables $U_i$ constructs an independent sample with distribution function $G$. Its empirical distribution function is $G_n(x)=F_n(G(x))$, where $F_n$ is the uniform-sample empirical distribution function. Consequently ${\sup }_x|G_n(x)-G(x)|\le {\sup }_{t\in [0,1]}|F_n(t)-t|$. For the uniform sample, monotonicity and a grid of mesh $1/m$ bound the last supremum by the largest error at grid points plus $1/m$. The strong law of large numbers on the countable union of all such grids proves the almost-sure limit. This construction proves the theorem for distributions with atoms as well as continuous distributions.

If $X_n$ converges in probability to both $X$ and $Y$, then $X=Y$ almost surely. Indeed, the triangle inequality and a union bound show that $\mathbb{P}(|X-Y|>\epsilon )=0$ for every $\epsilon >0$.

Convergence in probability is equivalent toThe forward implication splits at a fixed error threshold; the reverse implication is Markov's inequality.

From $X_n\to X$ in probability, choose $n_k$ with
$\mathbb{P}(|X_{n_k}-X|>2^{-k})<2^{-k}$. The first Borel--Cantelli lemma then gives $X_{n_k}\to X$ almost surely.

Independent indicators with success probabilities $1/n$ converge to zero in probability, but successes occur infinitely often almost surely by the second Borel--Cantelli lemma.