Affine invariance of Hotelling's statistic Created 2026-10-14 Updated 2026-11-05
Under a nonsingular affine transformation , transform the null mean by the same map. Then the centered sample mean becomes and the sample covariance matrix becomes . Since , the quadratic form in Hotelling's T-squared statistic is unchanged. Its value and test decision are consequently invariant under changes of units, origin and nonsingular coordinates.
Asymptotic distribution of a sample median Created 2026-09-24 Updated 2026-11-05
Let be independent and identically distributed random variables with unique median , and let their probability distribution have continuous probability density function with . For odd , the sample median satisfies
The influence function of the sample median has mean zero and variance : each of its two nonzero values occurs with probability . Its asymptotic linear representation, followed by the central limit theorem, gives the displayed limit.
For a standard normal distribution, , so the asymptotic variance of the unscaled sample median is . The sample mean instead has variance in that model. This quantifies an efficiency cost at the reference model without contradicting the sample median's bounded response to an outlier.
Autocovariance Created 2026-09-28 Updated 2026-10-14
The autocovariance measures how two observations of the same stochastic process vary together. For a real weakly stationary process of mean , it is
Weak stationarity removes dependence on the absolute time . Without it, the covariance function generally needs two time arguments. Lag zero is the variance, ; symmetry of real covariance gives . Positive values favor deviations from the mean in the same direction; negative values favor opposite directions. The units are the square of the observation units. See Bartlett, lecture 2: autocovariance examples.
For example, if with centered unit-variance white noise, expanding the product leaves only matching noise times. Thus , , and for . This MA(1) has a positive lag-one autocovariance because adjacent observations share one disturbance. Rescaling every observation by multiplies every autocovariance by , while leaving autocorrelation unchanged.
The autocovariance is useful beyond comparing pairs: it determines the variance of any finite linear combination of observations. For the sample mean ,
Each separation occurs in pairs. For the preceding MA(1) and , this is , compared with for four uncorrelated random variables of the same variance. This is the finite-record counterpart of the long-run variance of a stationary process.
Batch size 2026-11-05
Batch size counts the examples combined in a mini-batch. In the simple stochastic gradient descent loop where each group produces one parameter update, it controls how many examples contribute to that update. It differs from a training epoch, which counts a pass over the available training data.[ref]
For an original counting example, partition ten examples with nominal batch size and retain the remainder. The actual sizes are , giving updates per training epoch. Discarding the remainder instead gives two updates and uses only eight examples. More generally the retained-remainder update count is for examples, provided there is one update per group.
Batch size also changes gradient sampling noise. If independently sampled component gradients at a fixed parameter have scalar variance , their sample mean has
For an original exact example, suppose one component gradient is equally likely to be 0 or 2. Its mean is 1 and variance is 1. A two-example mini-batch gradient takes values with probabilities , giving variance . This calculation holds parameters fixed and assumes independent sampling; sampling without replacement has a finite population correction.
With gradient accumulation, several groups may contribute before an update. Two full four-example groups then involve eight examples per update if their gradients are normalized consistently. State whether a reported size counts the processed group or all examples contributing to an update. Lower sampling variance alone does not establish faster training for a fixed computation budget.
For a random sample with known and prior distribution with , compare to . The prior odds are one by symmetry. Normal-normal conjugacy gives a normal posterior distribution with
Its negative and nonnegative masses are and , where
Since the Bayes factor is the posterior odds divided by the prior odds,
Here is the cumulative distribution function of the standard normal distribution. The ratio decreases with the sample mean. For any threshold , the inequality is equivalent to and hence to
For , the quantile is , and the equivalent posterior probability threshold for is . The symmetry and proper normalization of the two hypothesis-conditional priors are essential to replacing the Bayes factor by the posterior odds directly.
Bayesian normal model Created 2026-09-24 Updated 2026-11-05
A Bayesian normal model combines a normal distribution for observations with a prior distribution for its unknown parameters. For independent with known and prior , normal-normal conjugacy gives posterior variance . The posterior mean is the corresponding precision-weighted average of and the sample mean. Unknown observation variance requires an additional prior; treating it as known yields different uncertainty. The model makes prior information and measurement variation explicit.
Let be fixed and known, and let be independent and identically distributed Bernoulli random variables with common success probability . Their joint probability mass function, on the binary sample space , is
There are arrangements with successes, so has the binomial distribution. For and , its exact conditional full-data probability mass function is
This conditional distribution is uniform over arrangements, with no dependence on : the sum is a sufficient statistic. [ref]
Figure 1.
Four Bernoulli observations grouped by their sufficient sum
. All sixteen binary sequences are grouped by . Within each group the conditional distribution is uniform; changing changes the group's total probability, not the conditional weights within it. Details.
For , the group sizes are and the within-group conditional probabilities are respectively . For example, leaves exactly , each with conditional probability . At , each such sequence has unconditional probability and the group has probability . At , these values are and ; at they return to and . The conditional weights stay in all three cases.
At the boundary , only has positive probability; at , only does. Dividing by on the other groups would divide by zero. Nevertheless, the same uniform-on-arrangements Markov kernel is a valid common version for the entire family : it gives the correct point mass on the one observed group at each boundary and assigns permissible versions on its unobserved groups. Define the joint probability mass function there by the original product of the one-trial Bernoulli probability mass functions, or use the limiting convention in the compact formula.
The sample mean is an unbiased estimator of and retains the sufficient statistic because is known. By contrast, is also an unbiased estimator of but, for , is not sufficient: the conditional success probability of the independent given is still . Thus unbiasedness does not imply sufficiency.
Sufficiency also does not imply completeness of a statistic. For , the full-data sufficient statistic admits the nonzero function with expectation zero for every . Nor does minimality imply completeness of a statistic: for and the restricted statistical model , is a minimal sufficient statistic since the likelihood ratio distinguishes its three values. Yet has expectation at both parameter values and is never zero. Changing the parameter family can change completeness of a statistic even while preserving sufficiency.
A categorical variable with two possible categories. Coding an event as one and its complement as zero makes the sample mean of that indicator variable equal to the event's sample proportion.
Bootstrapping (statistics) Created 2026-09-24 Updated 2026-10-18
The bootstrap approximates a sampling distribution by repeatedly sampling with replacement from the empirical measure of the observed data. In the ordinary nonparametric version, observations are assumed independent and identically distributed; a bootstrap sample has the same size as the original statistical sample. A statistic, such as a sample mean or sample median, is recomputed on each resample. Its variation across resamples estimates how the original statistic would vary across repeated sampling from the unknown probability distribution.[ref]
The key distinction is between the observed statistical sample and its still-unknown source probability distribution. Condition on the observed values and form the empirical measure
Here is a Dirac measure. Draw independent indices from the uniform distribution on a finite set and set . Each observed slot has probability on every draw. If equal values occupy several slots, their total probability is the sum of those slot probabilities; distinct observed values need not be equally likely. A star denotes this conditional resampling experiment. There are equally likely ordered index sequences, although bootstrap count vectors and statistics can combine many such sequences.
Figure 1.
Sampling with replacement reuses observed slots while keeping the original data fixed
. Original schematic for observations . Three possible ordered bootstrap samples have different sample means. Their colors identify source slots; none of these samples is new evidence about the unknown probability distribution. Details.
A complicated sampling distribution need not stop statistical inference when recomputing the statistic is easy. The bootstrap reuses that calculation on simulated datasets. The plug-in estimator principle replaces the unknown probability measure by in the whole repeated-sampling experiment. Whether that substitution approximates the target sampling distribution needs an appropriate bootstrap consistency theorem for the sample mean or a result for the particular statistic. A larger number of resamples reduces Monte Carlo error; it cannot repair the bootstrap failure for a uniform endpoint, an unrepresentative statistical sample or ignored statistical dependence.[ref]
Boundary likelihood-ratio test for two Gaussian means Created 2026-10-14 Updated 2026-11-05
For independent unit-variance Gaussian samples with means , testing against unrestricted means gives . The unrestricted maximum-likelihood estimators are the two sample means, while the null estimators are and zero. Under the null, write , , and , with independent standard Gaussian variables. Then decreases pointwise with , so the largest rejection probability occurs at the boundary . There its law is the chi-bar-squared distribution . The exact size- threshold is therefore determined by .
Breakdown point Created 2026-09-24 Updated 2026-11-05
A breakdown point measures how much contamination can make an estimator fail without bound. It specifies a contamination mechanism and a notion of failure. For scalar location under replacement contamination, failure means that replacing some observations can move the estimated location arbitrarily far.
The sample mean of observations can be made arbitrarily large by changing just one observation. For an odd sample of size , the sample median stays between original data values after at most replacements, but replacements can send it to infinity.
Two finite-sample conventions record either the first fraction allowing unbounded error or the largest fraction still guaranteed bounded. For the sample mean they are and ; for that odd-sample sample median they are and . The replacement breakdown point must state which convention is used. Both describe a finite change, whereas an influence function describes an infinitesimal perturbation.
Brian Brookman 2026-11-05
Brian Brookman works on analytical measurement quality and proficiency testing. His official 2026 Eurachem workshop biography describes him as an independent consultant and former Chief Scientific Officer of LGC's global proficiency-testing business. It records work on training, development of testing schemes and accreditation.[ref]
Figure 1.
Brian Brookman, portrait supplied on the named Eurachem workshop speaker page
.
With Ian Mann, Brookman co-edited the 2021 third edition of Selection, Use and Interpretation of Proficiency Testing (PT) Schemes by Brian Brookman and Ian Mann. The editorial page lists him as chair of its working group. The guide explains how laboratories select comparisons, interpret performance and investigate questionable results.[ref]
An original statistical example shows one reason scheme design matters. Four reported results have sample mean110 and sample standard deviation20. Assessing the value140 against that mean and spread gives . If an independently justified assigned value is100 and the required assessment spread is5, the same reported value gives a proficiency-test z-score of8. The participant values have not changed; the reference and criterion have.
This small invented dataset is not a prescription to estimate a real scheme from four participants. It illustrates how an extreme result can affect a participant-derived consensus and spread. The guide discusses assigned values, standard deviation for proficiency assessment and statistical evaluation as distinct design choices. A laboratory should understand those choices before interpreting a score as a simple accuracy ranking.[ref]
The scientific role of this work is to connect measurement accuracy, measurement uncertainty and externally defined performance criteria. A comparison can reveal a problem, but determining its cause still requires examination of the method, materials, measurement calibration and data handling.
Cauchy sample mean has an unchanged distribution Created 2026-10-14 Updated 2026-11-05
independent standard Cauchy distribution variables sum to a Cauchy law of scale , so their sample mean has its original distribution for every sample size. This follows by induction using convolution of probability densities. It does not contradict the usual weak law of large numbers, since the Cauchy law has no finite first absolute moment and its expectation is undefined. Symmetry does not turn a divergent expectation into zero.
Central limit theorem Created 2026-09-24 Updated 2026-10-14
Let be real independent and identically distributed random variables with expected value and finite, strictly positive variance . Write and for their sum and sample mean. The central limit theorem states
Precisely, for every real ,
where is the standard normal distribution function. The conclusion is convergence in distribution to the standard normal distribution. If the variance is zero, every observation equals almost surely; the displayed normalization is then undefined and there is no fluctuating error to describe.
The striking feature is that the observations need not have a normal distribution. They may be binary, bounded, or strongly skewed. Their detailed probability distribution disappears from this limit except through the expected value and variance. The subtraction removes predictable growth: . The division implements variance normalization: variance additivity for independent random variables gives , so every has expected value zero and variance one. For nonzero , an uncentered sum drifts; without rescaling its spread grows.
For a concrete example, let each have a Bernoulli distribution with success probability . Then has a binomial distribution, with expected value and variance . Its exact probability mass function and its normalized count are
Here is a binomial coefficient. Individual trials are conspicuously skewed, yet their centered, rescaled counts approach the same standard normal distribution as sums of fair trials.
Figure 1.
Exact standardized binomial distributions approach the standard normal curve
. For independent Bernoulli trials with , each blue bar has width equal to the normalized lattice spacing and area equal to the exact binomial distribution mass at its center. Heights are mass divided by spacing, so they can be compared with the orange probability density function of the standard normal distribution. All four normalized counts have expected value zero and variance one. The rectangles visualize discrete masses; they do not make the discrete probability distributions continuous. Details.
At , has expected value and standard deviation . The exact binomial distribution gives . A normal approximation with the continuity correction uses the interval to :
The small discrepancy is an approximation error, not an equality supplied by the central limit theorem.
The strong law of large numbers and weak law of large numbers locate the average: , respectively by almost sure convergence and convergence in probability. The central limit theorem magnifies that shrinking error and describes its shape. In terms of the sample mean, the resulting normal approximation is , with standard error . This explains why four times as many independent observations halve the standard error. When is known, has approximately coverage for at large , giving a confidence interval. This coverage refers to repeated samples, not a guarantee for an individual interval.
A useful application is the Monte Carlo method: estimate a success probability by the proportion of successful simulated trials. Nicholas Metropolis and S. Ulam's original paper, "The Monte Carlo Method" (1949), illustrates this with games of solitaire and estimation of a region's volume by randomly selected points, and discusses why estimates require probabilistic error control. The strong law of large numbers justifies convergence of such proportions for independent repetitions; the central limit theorem supplies their large-sample error scale when .[ref]
There are three useful limits to this picture:
A historical original-paper reference is J. W. Lindeberg's "Eine neue Herleitung des Exponentialgesetzes in der Wahrscheinlichkeitsrechnung" (1922), published in Mathematische Zeitschrift 15, pp. 211–225. The Lindeberg condition and Lindeberg-Feller central limit theorem below describe a version for sums whose summands need not share a probability distribution.[ref]
A brief proof outline uses the characteristic function of . Its expected value is zero and its variance is one, so the finite second moment gives
The Taylor expansion of the complex exponential, with its remainder controlled by , and the dominated convergence theorem justify this step without a third-moment assumption. Independence then gives the characteristic function of :
The limit is the characteristic function of the standard normal distribution; the Lévy continuity theorem turns this into convergence in distribution.
Let be real independent and identically distributed random variables, with mean , variance and finite fourth central moment . For the sample variance , the displayed convergence in distribution holds.
Indeed . The central limit theorem for the sample mean implies that tends to zero in probability. Changing the divisor from to also has negligible effect at this scale, by the law of large numbers. Apply the central limit theorem to the independent variables , whose variance is , and use Slutsky's theorem. Standardization gives a standard normal distribution limit when . The familiar special case holds for the normal distribution because then .
For a nonempty finite collection of Euclidean vectors, its sample mean uniquely minimizes the sum of squared Euclidean distances to a proposed centroid . Expand each squared Euclidean norm around and use to cancel the cross term. The displayed identity leaves a nonnegative term , equal to zero exactly at .
Characteristic function of a sum of independent variables Created 2026-09-24 Updated 2026-10-18
Independence makes the characteristic function of a sum equal the product of the individual characteristic functions.
For mutually independent random variables , write . Then
This is the expectation of a product of independent random variables applied to bounded complex exponentials; no moment hypothesis is needed.[ref]
Identical probability distributions are unnecessary. When the random variables are independent and identically distributed random variables, the product is . A deterministic rescaling also follows directly from the definition: for real . Consequently the sample mean has characteristic function , and a sum divided by has characteristic function .
The independence assumption does real work. Let take and with equal probabilities and set . Both characteristic functions are , but , not . At they give and zero respectively. Knowing the two marginal probability distributions does not determine the sum without their joint probability distribution.
Let be independent and identically distributed random variables taking and with equal probabilities, the Rademacher distribution. The two opposite complex exponentials give . Their expected value is zero and their variance is one. For , the characteristic functions of its sample mean and its variance normalization are respectively
For each fixed , the bases are positive for all sufficiently large . The Taylor expansion at zero then gives
Exponentiating proves the two displayed pointwise limits. The first limit is the characteristic function of the constant zero; the second is that of the standard normal distribution. The Lévy continuity theorem gives the corresponding convergence in distribution statements. Averaging has variance , whereas dividing the sum by preserves variance one. This sign calculation illustrates why the law of large numbers and the central limit theorem use different scales; it is not a proof for arbitrary probability distributions.
Chebyshev's inequality 2026-10-14
For a real random variable with finite variance , expected value , and a threshold , Chebyshev's inequality gives
One can also cap this upper bound at one, since a probability cannot exceed one. If , taking gives . For example, at least of the probability lies strictly within two standard deviations of the mean. This statement needs only the finite variance assumption. If the variance is zero, almost surely and every positive deviation threshold has zero tail probability.
Consider a random variable uniformly distributed on . Its expected value is zero and its variance is . At the actual tail probability is , while Chebyshev's inequality gives . At the actual tail probability is zero, while the bound is . A valid bound need not describe the exact shape or bounded support of a particular probability distribution.
Figure 1.
A general variance bound can be loose for a normal distribution and exact for another distribution
. Left: the two-sided tail probability of the standard normal distribution is compared with the capped Chebyshev's inequality bound. Right: the three-point probability distribution places probabilities , , at , , , giving variance one and probability at a deviation of at least two. The points at the threshold are included by . Details.
For the standard normal distribution, the actual tail probabilities beyond two and three standard deviations are approximately and , whereas the general bounds are and . More knowledge of the probability distribution can therefore improve the estimate. The comparison does not diminish the guarantee: the three-point example in equality in Chebyshev's inequality attains it exactly.
The proof is a one-line use of Markov's inequality on the nonnegative random variable . The event equals , and . This explains why the denominator is squared and why a finite variance supplies the necessary information. [ref].
One important use is control of a sample mean. For independent and identically distributed random variables with finite variance , the variance of a sample mean is . Hence
For every fixed tolerance, the bound tends to zero as the sample size grows, proving the finite-variance case of the weak law of large numbers. For and , makes the bound . This is a sufficient bound under the stated assumptions, rather than an optimal required sample size.
An early historical source is Pafnuty Chebyshev's 1867 paper “Des valeurs moyennes”, published in French translation by N. de Khanikof. Its opening theorem bounds deviations of sums using their expected values and second moments, linking averages with control of exceptional outcomes. [ref].
Cluster bootstrap 2026-11-05
A cluster bootstrap resamples entire observed groups, retaining all observations and their dependence within each selected group. The sampling unit is the group: draw group labels with replacement from the observed groups, concatenate their observations, and recompute the desired statistic. This is a form of bootstrapping suited to independent groups whose internal observations need not be independent. The Stata manual implements it with cluster(); idcluster() distinguishes two resampled copies of the same original group when the fitted procedure needs group identifiers.[ref]
For an original example, three sensors each supply two perfectly shared readings: , , and . The overall sample mean is . A cluster draw produces readings and sample mean ; the two copies of remain distinct resampling slots. Conditional on these data, the empirical variance of a group mean drawn uniformly from is . Averaging three independent draws therefore gives bootstrap variance
Resampling six individual rows instead gives
The row procedure halves the conditional variance by pretending that two identical within-sensor readings are independent information. These fractions use the empirical drawing law directly, not a Bessel-corrected estimate of an unknown population variance.
With unequal group sizes, drawing the same number of groups can produce a different number of rows in each replicate. A pooled row mean and a mean of group means are different statistics: unequal-cluster weighting in a cluster bootstrap must follow the intended estimand. Many rows from a few sensors do not replace having many independent sensors; arbitrary dependence between sensors, a dominant group, or too few groups can defeat the intended inference.