An autoregressive process of order one with independent and identically distributed random variables as innovations is an affine process when the innovation exponential moments are finite. Its one-step coefficients are and , where is the innovation cumulant-generating function.
Aggregate claims model Created 2026-10-06 Updated 2026-11-05
An aggregate claims model separates the number of claims from their sizes , with total and the empty sum equal to zero. Under independent and identically distributed random variables for claim sizes, independent of , conditional expectation gives tractable moments and transforms.
Asymptotic distribution of a sample median Created 2026-09-24 Updated 2026-11-05
Let be independent and identically distributed random variables with unique median , and let their probability distribution have continuous probability density function with . For odd , the sample median satisfies
The influence function of the sample median has mean zero and variance : each of its two nonzero values occurs with probability . Its asymptotic linear representation, followed by the central limit theorem, gives the displayed limit.
For a standard normal distribution, , so the asymptotic variance of the unscaled sample median is . The sample mean instead has variance in that model. This quantifies an efficiency cost at the reference model without contradicting the sample median's bounded response to an outlier.
Asymptotic equipartition property Created 2026-09-24 Updated 2026-10-14
The asymptotic equipartition property explains why a long message from a source can usually be described using its information entropy per symbol. Individual messages can have very different probabilities, but the messages carrying almost all the total probability have similar logarithmic costs per symbol. The Bernoulli typical-set example makes the distinction between an individual message and a large collection explicit.
For independent and identically distributed random variables on a finite alphabet, let be their probability mass function and . Then
Here means the probability mass of the realized word, evaluated under the source law. The product rule for independent random variables turns its information content into a sum:
The strong law of large numbers gives the limit. Zero-probability symbols do not occur; on the positive-probability support of a finite alphabet, these information costs are finite. The corresponding typical set consists of words whose cost per symbol is within a chosen tolerance of .
For a strictly stationary process with a finite alphabet, the constant-limit extension requires ergodicity. The Shannon-McMillan-Breiman theorem then replaces the one-symbol information entropy with the information rate. Stationarity without ergodicity in an information source shows why stationarity alone is insufficient.[ref]
Bernoulli process 2026-10-14
A Bernoulli process is a sequence of independent and identically distributed random variables with a Bernoulli distribution: and . It models repeated Bernoulli trials with a fixed success probability and no dependence between trials. The count has a binomial distribution, expected value , and variance ; the first success time has a geometric distribution on . The observations and the accumulated count are different stochastic processes: the observations are independent, while consecutive counts share all earlier observations. [ref]
Biased random walk Created 2026-09-24 Updated 2026-10-14
A biased random walk on the integer line takes unit steps right and left with unequal probabilities and , using independent and identically distributed random variables as increments. For , the expected value is : positive for and negative for . Changing to reverses the preferred direction. The exact endpoint probability mass function is given under simple random walk on the integer line.
For example, with each step has expected value and variance . After steps,
The displacement of the expected value from zero is about standard deviations, yet individual sample paths still move left as well as right, as the figure under simple random walk illustrates. Bias lets this elementary stochastic process model an average directional tendency while retaining fluctuations. Boundary and return questions have their own treatments under gambler's ruin and the specialized children below.
Catoni mean estimator Created 2026-09-28 Updated 2026-11-05
A Catoni mean estimator estimates an expected value from independent and identically distributed random variables with finite variance, using a nonlinear score instead of directly averaging the observations. Choose a continuous monotone function satisfying
and solve for .
For known variance , set , , and assume . The choices
give . The quadratic envelope controls exponential moments of the transformed score using only second moments; Markov's inequality and score monotonicity then bracket its root. The tuning depends on the variance and confidence level in this version.[ref]
The estimating score is not automatically a normalized population influence function. This procedure targets the mean of the sampled probability distribution; it does not identify an uncontaminated component's location without additional assumptions.
Central limit theorem Created 2026-09-24 Updated 2026-10-14
Let be real independent and identically distributed random variables with expected value and finite, strictly positive variance . Write and for their sum and sample mean. The central limit theorem states
Precisely, for every real ,
where is the standard normal distribution function. The conclusion is convergence in distribution to the standard normal distribution. If the variance is zero, every observation equals almost surely; the displayed normalization is then undefined and there is no fluctuating error to describe.
The striking feature is that the observations need not have a normal distribution. They may be binary, bounded, or strongly skewed. Their detailed probability distribution disappears from this limit except through the expected value and variance. The subtraction removes predictable growth: . The division implements variance normalization: variance additivity for independent random variables gives , so every has expected value zero and variance one. For nonzero , an uncentered sum drifts; without rescaling its spread grows.
For a concrete example, let each have a Bernoulli distribution with success probability . Then has a binomial distribution, with expected value and variance . Its exact probability mass function and its normalized count are
Here is a binomial coefficient. Individual trials are conspicuously skewed, yet their centered, rescaled counts approach the same standard normal distribution as sums of fair trials.
Figure 1.
Exact standardized binomial distributions approach the standard normal curve
. For independent Bernoulli trials with , each blue bar has width equal to the normalized lattice spacing and area equal to the exact binomial distribution mass at its center. Heights are mass divided by spacing, so they can be compared with the orange probability density function of the standard normal distribution. All four normalized counts have expected value zero and variance one. The rectangles visualize discrete masses; they do not make the discrete probability distributions continuous. Details.
At , has expected value and standard deviation . The exact binomial distribution gives . A normal approximation with the continuity correction uses the interval to :
The small discrepancy is an approximation error, not an equality supplied by the central limit theorem.
The strong law of large numbers and weak law of large numbers locate the average: , respectively by almost sure convergence and convergence in probability. The central limit theorem magnifies that shrinking error and describes its shape. In terms of the sample mean, the resulting normal approximation is , with standard error . This explains why four times as many independent observations halve the standard error. When is known, has approximately coverage for at large , giving a confidence interval. This coverage refers to repeated samples, not a guarantee for an individual interval.
A useful application is the Monte Carlo method: estimate a success probability by the proportion of successful simulated trials. Nicholas Metropolis and S. Ulam's original paper, "The Monte Carlo Method" (1949), illustrates this with games of solitaire and estimation of a region's volume by randomly selected points, and discusses why estimates require probabilistic error control. The strong law of large numbers justifies convergence of such proportions for independent repetitions; the central limit theorem supplies their large-sample error scale when .[ref]
There are three useful limits to this picture:
A historical original-paper reference is J. W. Lindeberg's "Eine neue Herleitung des Exponentialgesetzes in der Wahrscheinlichkeitsrechnung" (1922), published in Mathematische Zeitschrift 15, pp. 211–225. The Lindeberg condition and Lindeberg-Feller central limit theorem below describe a version for sums whose summands need not share a probability distribution.[ref]
A brief proof outline uses the characteristic function of . Its expected value is zero and its variance is one, so the finite second moment gives
The Taylor expansion of the complex exponential, with its remainder controlled by , and the dominated convergence theorem justify this step without a third-moment assumption. Independence then gives the characteristic function of :
The limit is the characteristic function of the standard normal distribution; the Lévy continuity theorem turns this into convergence in distribution.
Let be real independent and identically distributed random variables, with mean , variance and finite fourth central moment . For the sample variance , the displayed convergence in distribution holds.
Indeed . The central limit theorem for the sample mean implies that tends to zero in probability. Changing the divisor from to also has negligible effect at this scale, by the law of large numbers. Apply the central limit theorem to the independent variables , whose variance is , and use Slutsky's theorem. Standardization gives a standard normal distribution limit when . The familiar special case holds for the normal distribution because then .
Characteristic function of a sum of independent variables Created 2026-09-24 Updated 2026-10-18
Independence makes the characteristic function of a sum equal the product of the individual characteristic functions.
For mutually independent random variables , write . Then
This is the expectation of a product of independent random variables applied to bounded complex exponentials; no moment hypothesis is needed.[ref]
Identical probability distributions are unnecessary. When the random variables are independent and identically distributed random variables, the product is . A deterministic rescaling also follows directly from the definition: for real . Consequently the sample mean has characteristic function , and a sum divided by has characteristic function .
The independence assumption does real work. Let take and with equal probabilities and set . Both characteristic functions are , but , not . At they give and zero respectively. Knowing the two marginal probability distributions does not determine the sum without their joint probability distribution.
Let be independent and identically distributed random variables taking and with equal probabilities, the Rademacher distribution. The two opposite complex exponentials give . Their expected value is zero and their variance is one. For , the characteristic functions of its sample mean and its variance normalization are respectively
For each fixed , the bases are positive for all sufficiently large . The Taylor expansion at zero then gives
Exponentiating proves the two displayed pointwise limits. The first limit is the characteristic function of the constant zero; the second is that of the standard normal distribution. The Lévy continuity theorem gives the corresponding convergence in distribution statements. Averaging has variance , whereas dividing the sum by preserves variance one. This sign calculation illustrates why the law of large numbers and the central limit theorem use different scales; it is not a proof for arbitrary probability distributions.
Chebyshev's inequality 2026-10-14
For a real random variable with finite variance , expected value , and a threshold , Chebyshev's inequality gives
One can also cap this upper bound at one, since a probability cannot exceed one. If , taking gives . For example, at least of the probability lies strictly within two standard deviations of the mean. This statement needs only the finite variance assumption. If the variance is zero, almost surely and every positive deviation threshold has zero tail probability.
Consider a random variable uniformly distributed on . Its expected value is zero and its variance is . At the actual tail probability is , while Chebyshev's inequality gives . At the actual tail probability is zero, while the bound is . A valid bound need not describe the exact shape or bounded support of a particular probability distribution.
Figure 1.
A general variance bound can be loose for a normal distribution and exact for another distribution
. Left: the two-sided tail probability of the standard normal distribution is compared with the capped Chebyshev's inequality bound. Right: the three-point probability distribution places probabilities , , at , , , giving variance one and probability at a deviation of at least two. The points at the threshold are included by . Details.
For the standard normal distribution, the actual tail probabilities beyond two and three standard deviations are approximately and , whereas the general bounds are and . More knowledge of the probability distribution can therefore improve the estimate. The comparison does not diminish the guarantee: the three-point example in equality in Chebyshev's inequality attains it exactly.
The proof is a one-line use of Markov's inequality on the nonnegative random variable . The event equals , and . This explains why the denominator is squared and why a finite variance supplies the necessary information. [ref].
One important use is control of a sample mean. For independent and identically distributed random variables with finite variance , the variance of a sample mean is . Hence
For every fixed tolerance, the bound tends to zero as the sample size grows, proving the finite-variance case of the weak law of large numbers. For and , makes the bound . This is a sufficient bound under the stated assumptions, rather than an optimal required sample size.
An early historical source is Pafnuty Chebyshev's 1867 paper “Des valeurs moyennes”, published in French translation by N. de Khanikof. Its opening theorem bounds deviations of sums using their expected values and second moments, linking averages with control of exceptional outcomes. [ref].
Chernoff information Created 2026-09-24 Updated 2026-10-14
Chernoff information measures how quickly repeated independent evidence can distinguish two fixed probability distributions. Suppose a statistical hypothesis test observes independent and identically distributed random variables, with common law under one simple hypothesis and under the other. Give the two hypotheses fixed positive prior probabilities, and let be the smallest achievable average error probability. For distinct positive probability mass functions on a common finite alphabet,
Here is the natural logarithm, so the error exponent is measured in nats per observation. This optimal rate is unchanged by choosing different fixed positive prior probabilities; the best decision rule and finite-sample error can still change.[ref]
The interpretation is : a larger Chernoff information means fewer observations are needed at the leading exponential scale. It does not assert the exact finite-sample identity . The finite-sample Chernoff error bound and the explicit Chernoff information for complementary Bernoulli distributions make this distinction concrete. This is useful when designing repeated measurements or comparing sensors whose output probability distributions differ under two candidate conditions.
For probability mass functions with zero entries, define the affinity for and use , taking endpoint limits when needed. This avoids an ambiguous convention. Identical laws give ; disjoint probability supports give , because one observation separates them perfectly. The displayed finite-rate theorem concerns fixed laws with common positive support, not laws that change with or arbitrarily dependent repeated observations.
A compound mixed Poisson distribution is a random sum of independent claims whose count has a Poisson mixture law: conditional on a nonnegative random intensity , has Poisson distribution with parameter . The claim sizes are independent and identically distributed random variables, independent of the intensity and count mechanism. If is their moment-generating function, then wherever finite. A gamma distribution intensity of shape and rate gives a negative binomial distribution count with shape and success probability . Independent intensities with common gamma rate add their shapes by additivity of independent gamma distributions with a common rate.
For independent and identically distributed random variables that are integrable random variables, with , symmetry gives . The decreasing sigma-algebras generated by these future sums permit application of the reverse martingale convergence theorem.
Covariance of two components of a compound Poisson sum Created 2026-10-14 Updated 2026-11-05
Let be independent of independent and identically distributed random variables with finite second moments. Conditioning on gives covariance and conditional expectations . Law of total covariance and yield the formula. For per-claim excess of loss reinsurance payouts and , it becomes . The insurer and reinsurer aggregates share their claim count and are generally dependent.
Critical threshold for level-set percolation Created 2026-09-24 Updated 2026-11-05
For a random field on a countable locally finite graph, declare open when . Increasing can only remove open vertices, so the event that an infinite percolation cluster exists is decreasing in . One convention for the critical threshold for level-set percolation is
The value may be infinite, and the definition does not decide what happens at . On a general graph, a specified-root threshold can differ from a global existence threshold, so the convention matters.
For independent and identically distributed random variables with uniform distribution on , the open probability is for . The resulting model is site percolation, and reversing the parameter gives , where is its percolation critical probability. This example separates the direction of the level threshold from the usual increasing opening-density parameter.