Statistical inference

Randomized response masks which of two complementary binary questions a person answered. Suppose the sensitive property has probability

p

, and independently the direct question is selected with known probability

x

, while the complementary question is selected otherwise. Truthful answers then have yes probability

q = x p + (1 - x) (1 - p)

. The investigator observes only the answer, preserving uncertainty about an individual's status while permitting population statistical estimation when

x \neq = 1/2

For

n

independent responses,

Y \sim Bin (n, q)

, and the affine estimator

p = (Y / n - 1 + x) / (2 x - 1)

is an unbiased estimator. Its variance is

q (1 - q) / [n (2 x - 1)^{2}]

, by the mean and variance of a binomial distribution. Equivalently the variance is

p (1 - p) / n + x (1 - x) / [n (2 x - 1)^{2}]

, exposing the extra sampling uncertainty caused by the masking. At

x = 1/2

, all

p

produce

q = 1/2

and identifiability fails. At

x = 0

1

, there is no masking and the extra variance vanishes.

Because

0 \leq p \leq 1

, maximum likelihood estimation gives

p = min (1, max (0, p))

. This follows by maximizing the binomial likelihood over the interval

q \in [min (x, 1 - x), max (x, 1 - x)]

and then inverting its monotone affine parametrization. Clipping can introduce bias: for

n = 1

x = 3/4

and

p = 0

, the clipped estimate has mean

1/4

Exploratory analysis

Analysis investigating patterns and generating or refining hypotheses through the data. It is scientifically useful when its data-dependent role is reported honestly. Findings developed through exploration need further independent evaluation before being presented as confirmed predictions.

Confirmatory analysis

Analysis evaluating a hypothesis or model whose relevant specification was fixed independently of the data used for its evaluation. The evidential interpretation depends on the complete analysis procedure, including selection and stopping rules.

Convenience sampling

Recruitment based mainly on accessibility rather than known probability selection from a defined target population. A larger convenience sample can reduce some sampling variability without eliminating selection bias or making respondents relevant to the intended construct.

Convenience sample

The observations or participants obtained through convenience sampling.

Target population

The set of units or situations to which a study intends its claims to apply. A recruitment pool, the observed sample and the target population can differ. External validity concerns whether the evidence transports to the intended target.

Pseudolikelihood ( $\prod_{i} p (x_{i} ∣ x_{- i})$ )

A pseudolikelihood multiplies conditional probabilities rather than the full joint likelihood function. It can be useful for fitting a Markov random field, but using the product itself as a probability density defines a different model unless an additional equivalence is established.

Winner's curse

Overestimation of effect magnitude when noisy discoveries are selected for unusually strong apparent effects. Independent replication and explicit modeling of the selection process help correct the bias.

Frequency estimation ( $\hat{f}$ )

Frequency estimation infers a periodic signal's frequency from observed samples. It may use a peak of the discrete Fourier transform, or fit a sinusoidal model with amplitude, phase, and frequency as unknown parameters. Its attainable precision depends on noise, observation duration, sampling rate, and the estimator. Fourier frequency resolution alone does not specify that precision.

Effect size

An effect size measures the magnitude of an observed or modeled relationship. A standardized mean difference expresses a mean contrast in standard-deviation units; other effect sizes include a correlation coefficient or odds ratio. Magnitude is distinct from the evidence against a null hypothesis supplied by a p-value: a small effect size can have a small p-value in a sufficiently large sample.

Eta squared ( $η^{2} = S S_{effect} / S S_{total}$ )

The fraction of total observed sum of squares attributed to a model effect under a specified analysis of variance decomposition. Its interpretation depends on the design and sum-of-squares convention.

Partial eta squared ( $η_{p}^{2} = S S_{effect} / (S S_{effect} + S S_{error})$ )

An effect size comparing an effect sum of squares with that effect's own error sum of squares. It need not be the fraction of total observed variation, and partial values for several effects need not sum to one. The relevant error term must be identified, especially in repeated-measures designs.

Statistical process control

Statistical process control compares observed process outcomes with a stable reference model and control limits reflecting expected sampling variation. An observation outside those limits is a signal to investigate, not proof of a specific cause. Varying sample volumes require varying limits, and multiple testing changes the chance of a chance signal somewhere in the display.

CUSUM ( $C_{t} = max (0, C_{t - 1} + W_{t})$ )

A CUSUM accumulates evidence for a change in a monitored probability distribution. With

W_{t}

a log-likelihood ratio increment favoring a specified changed distribution, reset the accumulated score at zero and signal when it exceeds a chosen boundary

h

. For

C_{0} = 0

C_{t}

is the largest nonnegative sum of successive increments ending at

t

. This permits repeated detection of a change with an unknown onset, rather than a single fixed-start test.

Poisson likelihood-ratio CUSUM ( $W_{t} = Y_{t} lo g θ - (θ - 1) λ_{0 t}$ )

For independent Poisson observations, comparing mean

λ_{0 t}

with

θ λ_{0 t}

gives the displayed log-likelihood ratio. Updating a CUSUM with these scores targets the specified multiplicative increase. Its expected score is negative under the null and positive under the alternative when

θ > 1

, so resetting at zero discards earlier evidence against an increase.

Average run length ( $ARL = E in f {t \geq 1 : C_{t} \geq h}$ )

The average run length of a CUSUM is the expectation of its first signalling time under a specified data model and initial chart state. A large in-control average run length discourages false alarms; a small out-of-control value indicates rapid detection. The mean alone does not give a finite-horizon Type I error probability or the complete detection-delay distribution.

Fast initial response CUSUM ( $0 < C_{0} < h$ )

A fast initial response CUSUM starts a CUSUM with a positive head start below its signalling boundary. Before its first reset, it accumulates a fixed-start log-likelihood ratio like a sequential probability ratio test; after resetting to zero it behaves as an ordinary CUSUM. It gives faster initial detection when the process may already be out of control, at the cost of changed initial false-alarm behavior.

Hospital mortality funnel plot ( $(n_{i}, p_{i})$ )

A hospital mortality funnel plot places each hospital's observed mortality proportion against its case volume. Under an independent binomial distribution model with common target p, the standard error is

p (1 - p) / n_{i}

, so uncertainty bands contract as volume grows. This is a provider-performance plot, distinct in purpose and axes from a meta-analysis funnel plot. A signal requires investigation of data quality and patient case mix before drawing a causal conclusion.

Binomial funnel control limits ( $p_{0} \pm z_{1 - α /2} p_{0} (1 - p_{0}) / n$ )

The displayed pointwise control limits follow from the normal approximation to an observed binomial proportion. Clip plotted limits to the probability range. Their nominal coverage is approximate and can be poor for small expected counts; exact binomial tails give a discrete alternative. Pointwise coverage does not provide simultaneous coverage of all hospitals; use Bonferroni correction or another familywise error rate procedure when that is the required guarantee.

Control limits

Control limits bound the sampling outcomes expected under a specified reference process. An observation outside a limit is a statistical hypothesis testing signal to investigate the reference model. These are limits for observations under that model, rather than a confidence interval constructed around each observed estimate. Their calibration may be pointwise, simultaneous or sequential.

Jackknife resampling

The delete-one form of jackknife resampling recalculates a statistic after deleting each observation in turn. It uses one observed statistical sample to investigate how sensitive an estimator is to its individual observations, and supplies both a jackknife variance estimator and jackknife bias correction. The calculations are deterministic once the data are fixed: no random draws are needed.

In the basic setting,

X_{1}, \dots, X_{n}

are independent and identically distributed random variables, with sample size

n > 1

. Let

x_{1}, \dots, x_{n}

be the observed values and

θ_{n} = t_{n} (x_{1}, \dots, x_{n})

a scalar estimator of an estimand

θ

. Apply the same estimation rule to each reduced statistical sample, using its new sample size

n - 1

wherever the rule requires it:

θ_{(- i)} = t_{n - 1} (x_{1}, \dots, x_{i - 1}, x_{i + 1}, \dots, x_{n}), i = 1, \dots, n, \overline{θ}_{J} = \frac{1}{n} \sum_{i = 1}^{n} θ_{(- i)} .

Every index is deleted exactly once; an unusual value is treated by the same rule as every other value. The mean

\overline{θ}_{J}

summarizes the deletion estimates. They overlap in

n - 2

observations, so they are generally not independent random variables representing fresh repetitions of the original experiment. Cosma Shalizi's ^[ref] explain this construction and its scaling.

For a concrete sample mean, use the data

(1, 2, 3, 4, 10)

, whose full sample mean is

\overline{x} = 4

. The five deletion sample means, in observation order, are

(θ_{(- 1)}, \dots, θ_{(- 5)}) = (4.75, 4.5, 4.25, 4, 2.5), \overline{θ}_{J} = 4.

Deleting

1

raises the sample mean by

0.75

, while deleting

10

lowers the sample mean by

1.5

. Thus

10

has the largest deletion effect in this statistical sample. This is a descriptive sensitivity calculation, not a rule for removing that observation: the full sample mean remains

4

Figure 1.
Delete each observation once and recompute the sample mean
. The left panel shows all five reduced statistical samples from $(1, 2, 3, 4, 10)$ ; each crossed entry is omitted only for that row. The right panel shows their sample means against the full sample mean $4$ . The orange row makes the deletion effect of $10$ visible. **Details**.

Jackknife variance estimator ( $V_{J}$ )

The jackknife variance estimator estimates the sampling variance of the full-data estimator, using the deletion estimates and their mean from jackknife resampling:

V_{J} = \frac{n - 1}{n} \sum_{i = 1}^{n} (θ_{(- i)} - \overline{θ}_{J})^{2}, SE_{J} = V_{J} .

The square root is the estimated standard error. The factor

n - 1

multiplies the divisor-

n

variance of the deletion estimates: those estimates vary less than estimates from new independent experiments because almost all their observations are shared. Ryan Tibshirani's ^[ref] give the same scaling in terms of the divisor-

(n - 1)

sample variance of the deletion estimates.

For the sample mean, this formula has an exact algebraic simplification. With

s_{n}^{2}

denoting the usual sample variance,

θ_{(- i)} = \frac{n x - x _{i}}{n - 1}, \overline{θ}_{J} = \overline{x}, θ_{(- i)} - \overline{θ}_{J} = - \frac{x _{i} - x}{n - 1},

and therefore

V_{J} = \frac{1}{n ( n - 1 )} \sum_{i = 1}^{n} (x_{i} - \overline{x})^{2} = \frac{s _{n}^{2}}{n} .

Under the independent and identically distributed assumption and a finite second moment,

s_{n}^{2}

is an unbiased estimator of the population variance

σ^{2}

. Consequently

V_{J}

is an unbiased estimator of the sample mean's sampling variance

σ^{2} / n

. The equality

V_{J} = s_{n}^{2} / n

itself holds for any observed data with

n > 1

; its sampling interpretation uses the stated assumptions.

For

(1, 2, 3, 4, 10)

, the sum of squared deviations from the sample mean is

50

, giving sample variance

s_{5}^{2} = 50/4 = 12.5

. The deletion sample means have squared deviations totaling

3.125

, so

V_{J} = \frac{4}{5} (3.125) = 2.5 = \frac{12.5}{5}, SE_{J} = 2.5 \approx 1.58114.

(4)

Here

12.5

estimates the variance of individual observations, while

2.5

estimates the sampling variance of their sample mean. These answer different questions.

For more general statistics, validity requires regularity beyond the deletion formula, such as a sufficiently smooth function of sample means with suitable moments. Ordinary delete-one jackknife resampling can fail for nonsmooth quantiles, including the median; see Larry Wasserman's ^[ref]. With statistical dependence, deleting individual observations need not reproduce the relevant sampling variance; the deletion scheme must reflect that statistical dependence. Shalizi's ^[ref] distinguish the basic independent and identically distributed case from extensions.

Jackknife and bootstrap variance of a sample second moment ( $v_{J} = \frac{1}{n ( n - 1 )} \sum_{i} (x_{i}^{2} - \overline{x^{2}})^{2}$ )

For independent identically distributed observations with finite fourth moment, the sample mean of their squares has sampling variance

Var (X^{2}) / n

. Its jackknife variance estimator is exactly the displayed unbiased estimator, since its leave-one-out values differ from their average by

- (x_{i}^{2} - \overline{x^{2}}) / (n - 1)

. The conditional bootstrap variance of a sample mean is

v_{B} = n^{- 2} \sum_{i} (x_{i}^{2} - \overline{x^{2}})^{2} = (n - 1) v_{J} / n

. Thus the two methods have a known finite-sample scaling difference, despite sharing the same large-sample target.

Fisherian conditional inference

In a parametric statistical model, the likelihood function compares parameter values, while sampling distributions calibrate tests and confidence intervals. A sufficient statistic retains the parameter information. Conditioning on an observed ancillary statistic restricts that calibration to experiments with the same ancillary configuration, instead of averaging over configurations whose distribution is unrelated to the parameter. Since the ancillary's marginal law is parameter-free, conditional and unconditional likelihoods have the same parameter-dependent factor. With a nuisance parameter, a conditional law or pivotal quantity must remove it before inference for the parameter of interest can use a nuisance-free reference law.

Conditionality principle

When an experiment's configuration is observed, inference conditions on the experiment actually performed rather than averaging over its possible configurations. Conditioning on an ancillary statistic is a central application; freedom from a nuisance parameter must still be checked in the resulting conditional distribution.

Kolmogorov-Smirnov test

The one-sample Kolmogorov-Smirnov test checks goodness of fit by comparing an empirical distribution function

F_{n}

with a specified cumulative distribution function

F_{0}

. Its Kolmogorov-Smirnov statistic

D_{n}

is the largest vertical gap anywhere between the two curves. Every threshold asks the same concrete question: what proportion of the observations are at or below it, and what probability does the proposed probability distribution assign there? A large discrepancy supplies evidence against the null hypothesis.

For example, four stipulated observations

(1/10, 2/5, 4/5, 9/10)

compared with the uniform distribution on

[0, 1]

give

D_{4} = 3/10

. Just before

4/5

, only two observations have been counted, so the empirical distribution function approaches

1/2

, while

F_{0}

approaches

4/5

. The Kolmogorov-Smirnov statistic section calculates every jump-side gap. These are exact illustrative numbers, not experimental measurements or a claim of statistical significance.

The ordinary one-sample calibration assumes independent and identically distributed real observations and a simple hypothesis

H_{0} : F = F_{0}

, with

F_{0}

a continuous function fixed before inspecting these observations. All statistical parameters of

F_{0}

must be specified.^[ref] Under that null hypothesis, the probability integral transform gives

U_{i} = F_{0} (X_{i})

with the uniform distribution on

(0, 1)

, retaining independence. Consequently the sampling distribution of

D_{n}

depends on the sample size

n

, but not on the particular continuous

F_{0}

. This is the useful sense in which the calibration is distribution-free.

For the exponential distribution of rate one,

F_{0} (x) = 1 - e^{- x}

for

x \geq 0

. Take

x_{i} = - lo g (1 - u_{i})

for the four stipulated

u_{i}

above. The probability integral transform changes the horizontal coordinates but preserves both the order statistics and their cumulative heights, so the same Kolmogorov-Smirnov statistic

3/10

appears on both scales. More generally, for continuous

F_{0}

, the jump-side formulas depend only on

F_{0} (X_{(i)})

. Strict increase everywhere is unnecessary: intervals on which

F_{0}

is constant have probability zero under the null hypothesis. This explains the invariance without assuming a strictly positive probability density function.

Figure 1.
The probability integral transform preserves the Kolmogorov-Smirnov gap on exponential and uniform scales
. The same four stipulated points are shown as $x_{i} = - lo g (1 - u_{i})$ on the left and $u_{i} \in {1/10, 2/5, 4/5, 9/10}$ on the right. Open circles indicate left limits of the empirical distribution function, filled circles its values. The gap $3/10$ is approached from below the third observation on both scales. **Details**.

The observed Kolmogorov-Smirnov statistic is a discrepancy, whereas its p-value is a tail probability under the null hypothesis:

p = Pr_{H_{0}} (D_{n}^{new} \geq d_{obs}) .

Here

D_{n}^{new}

is computed from a fresh sample of the same sample size. Choose a significance level

α

before the test and reject when the calibrated p-value is at most

α

. The value

D_{4} = 3/10

alone supplies neither a p-value nor a rejection decision. Failure to reject does not establish that the null hypothesis is true. A finite-sample calibration and the asymptotic Kolmogorov-Smirnov theorem are distinct; a cutoff for

D_{n}

must be rescaled before use with

n D_{n}

.^[ref]

Fitting statistical parameters on the tested observations replaces

F_{0}

by the random

F_{θ}

. The resulting transformed observations need not be independent and identically distributed uniforms, so the ordinary reference law fails. The Lilliefors test addresses a particular fitted normal distribution. More general parametric bootstrap calibration must simulate from the fitted statistical model and repeat the fitting in every replicate; holding the fitted statistical parameters fixed during those comparisons calibrates a different procedure. Such a parametric bootstrap requires its own validity justification rather than inheriting exactness from the simple-null test.

For a discrete probability distribution,

F_{0} (X)

is generally not uniform. Equal observations also arise from rounding, so tied data require a calibration that accounts for the discrete probability distribution or the rounding rule. The discrepancy remains meaningful, but continuous-null p-values are not automatically valid. Statistical dependence likewise changes the reference law. The two-sample Kolmogorov-Smirnov test compares two empirical distribution functions and has a different calibration.^[ref]

Kolmogorov-Smirnov statistic ( $D_{n} = sup_{x} ∣ F_{n} (x) - F (x) ∣$ )

For an empirical distribution function

F_{n}

and a specified cumulative distribution function

F_{0}

, the one-sample Kolmogorov-Smirnov statistic is their Kolmogorov distance:

F_{n} (x) = \frac{1}{n} \sum_{j = 1}^{n} 1_{{X_{j} \leq x}}, D_{n} = sup_{x \in R} ∣ F_{n} (x) - F_{0} (x) ∣.

The displayed sum uses an indicator function. With this convention,

F_{n}

is right-continuous. For distinct observations, its value and its left one-sided limit at the

i

th order statistic are

F_{n} (X_{(i)}) = i / n

and

F_{n} (X_{(i)}^{-}) = (i - 1) / n

For continuous

F_{0}

, both jump sides matter. Define the two directional discrepancies by

D_{n}^{+} = sup_{x} (F_{n} (x) - F_{0} (x)) = max_{1 \leq i \leq n} {\frac{i}{n} - F_{0} (X_{(i)})},

D_{n}^{-} = sup_{x} (F_{0} (x) - F_{n} (x)) = max_{1 \leq i \leq n} {F_{0} (X_{(i)}) - \frac{i - 1}{n}}, D_{n} = max (D_{n}^{+}, D_{n}^{-}) .

Thus

D_{n}^{+}

uses the empirical distribution function immediately after a jump, and

D_{n}^{-}

uses its left one-sided limit.^[ref] A supremum need not be attained.

For

n = 4

F_{0} (x) = x

[0, 1]

, and the stipulated order statistics

(1/10, 2/5, 4/5, 9/10)

, the complete calculation is

i 1234 X_{(i)} 1/10 2/5 4/5 9/10 F_{4} (X_{(i)}^{-}) 0 1/4 1/2 3/4 F_{4} (X_{(i)}) 1/4 1/2 3/4 1 i /4 - F_{0} (X_{(i)}) 3/20 1/10 - 1/20 1/10 F_{0} (X_{(i)}) - (i - 1) /4 1/10 3/20 3/10 3/20

(4)

The last two columns give the signed directional gaps:

D_{4}^{+} = 3/20

D_{4}^{-} = 3/10

, and

D_{4} = 3/10

. In contrast, checking only

∣ F_{4} (X_{(i)}) - F_{0} (X_{(i)}) ∣

gives

3/20

and misses the largest gap approached as

x ↑ 4/5

. At

x = 4/5

itself, the gap has already fallen to

1/20

Figure 1.
Both jump sides determine the exact Kolmogorov-Smirnov statistic
. The empirical distribution function for four stipulated rational observations is compared with the uniform distribution on $[0, 1]$ . Open circles show left one-sided limits; filled circles show values at jumps. The green gap is $D_{4}^{+} = 3/20$ and the magenta gap is the limiting $D_{4}^{-} = 3/10$ . **Details**.

The finite formula follows from the behavior between jumps: the empirical distribution function

F_{n}

is constant and the cumulative distribution function

F_{0}

is monotone, so the largest directional gaps occur at the two interval ends, possibly as one-sided limits. The two tails introduce no larger gap. Checking every jump side therefore gives the exact supremum.

A common normalized Kolmogorov-Smirnov statistic is

n D_{n}

, which equals

3/5

in this example. Its large-sample reference law comes from the Kolmogorov-Smirnov theorem. The unscaled

D_{n}

remains the largest difference of cumulative probabilities, not a p-value.

There is a useful exact exception to the usual warning about fitting statistical parameters. Suppose

n \geq 2

independent and identically distributed observations have the uniform distribution on

(0, θ)

with unknown

θ > 0

. Let

M = max_{i} X_{i}

and let

J

be the label of its almost surely unique maximizing observation. Conditional on

M = m

and

J = j

, the surviving labelled observations

(X_{i})_{i \neq = j}

are independent with the uniform distribution on

(0, m)

; after division by

m

they are independent and identically distributed uniforms on

(0, 1)

. These are conditional distributions defined by probability density functions, since the event

M = m

has probability zero.

To check this, the joint probability density function of

M

and

J = j

m^{n - 1} / θ^{n}

for

0 < m < θ

. Dividing the original joint probability density function

θ^{- n}

by this factor gives the conditional survivor probability density function

m^{- (n - 1)} \prod_{i \neq = j} 1_{{0 < x_{i} < m}} .

(5)

This factorization proves the claimed conditional independence; scaling removes

m

. Keep the surviving labels in a fixed order, or apply an independent random permutation. The sorted survivors instead have the joint law of uniform order statistics and exhibit statistical dependence; sorting does not affect their empirical distribution function. Delete the maximum itself, which would otherwise become a forced value

1

, and compute the Kolmogorov-Smirnov statistic using sample size

n - 1

and jump heights

i / (n - 1)

. This gives an exact conditional Kolmogorov-Smirnov test under the uniform-endpoint statistical model. It does not justify fitting an arbitrary endpoint or other statistical parameters and using the ordinary

n

-sample calibration.

Lilliefors test ( $sup_{x} ∣ F_{n} (x) - Φ ((x - \overline{X}) / s) ∣$ )

The Lilliefors test calibrates the Kolmogorov-Smirnov statistic against a normal distribution whose mean and variance are estimated from the tested observations. In the original procedure,

s^{2}

is the sample variance with denominator

n - 1

, and

Φ

is the standard normal distribution function. Its reference law includes this fitting step. The original paper's opening procedure and Table 1 explain the changed calibration.^[ref]

Two-sample Kolmogorov-Smirnov test ( $D_{n, m} = sup_{x} ∣ F_{n} (x) - G_{m} (x) ∣$ )

The two-sample Kolmogorov-Smirnov test compares two empirical distribution functions from independent samples, each internally independent and identically distributed, under the null hypothesis that their underlying probability distributions coincide. Both curves are estimated from data, so calibration depends on both sample sizes; the usual asymptotic scaling is

nm / (n + m) D_{n, m}

. Tied data can be handled by an appropriate permutation test under the required exchangeability, rather than by assuming the untied continuous-null law.^[ref]

Likelihood principle

All evidence in observed data about a parameter is contained in its likelihood function; proportional likelihoods carry the same evidence. If a stopping rule contributes only a positive factor independent of the parameter to the complete observed-sample likelihood, that factor does not change likelihood-based inference. Recording only a coarsened part of the sample produces a different likelihood and is not covered by the same invariance claim.

Exponential sampling stopped by a data-dependent coin ( $L (θ; x) \propto θ^{N} e^{- θ \sum X_{i}}$ )

For independent exponential observations of rate

θ

, stopping after observation

x

with known probability

α (x) \in (0, 1)

gives a complete-data likelihood proportional to

θ^{N} e^{- θS}

. Its minimal sufficient statistic is

(N, S)

by the parameter-free likelihood-ratio criterion. If only the last value

Y

is recorded, its density instead is

α (y) e^{- θ y} / \int_{0}^{\infty} α (x) e^{- θ x} d x

. The stopping rule therefore changes the coarsened-data inference even though it adds no complete-data likelihood factor involving

θ

Estimand ( $ψ (P)$ )

An estimand is the population quantity an analysis seeks to learn, as distinct from an estimator calculated from a sample. In a randomized controlled trial, the causal effect of treatment assignment and the effect of actual treatment receipt are different estimands when adherence is imperfect.

Efficiency (statistics)

Statistical efficiency compares the precision of procedures estimating the same estimand. For unbiased estimators, relative efficiency is commonly a ratio of their variances; lower variance means more information from the same sample size. An efficient estimator attains a relevant information lower bound. A baseline-adjusted linear regression can improve precision in a randomized controlled trial by removing prognostic variation.

Statistical functional ( $ψ : P \to R$ )

A statistical functional assigns a target value to each probability measure in a statistical model. Examples include an expected value, a quantile, and an integral of a power of a probability density function. Its behavior along statistical paths determines whether first-order influence-function representers are available.

Functional statistic ( $T (F_{n})$ )

A functional statistic evaluates a statistical functional

T

at the empirical distribution

F_{n} = n^{- 1} \sum_{i} δ_{X_{i}}

. The map

T

acts on distributions, whereas

T (F_{n})

is a statistic. If

T

has an asymptotic linear representation, then

T (F_{n}) - T (F) = \frac{1}{n} \sum_{i} IF (X_{i}; T, F) + o_{p} (n^{- 1/2}) .

A square-integrable influence function with mean zero consequently gives asymptotic variance

n^{- 1} \int IF (x; T, F)^{2} d F (x)

by the central limit theorem.

Density fourth-power functional ( $ψ (f) = \int f^{4}$ )

Along bounded density tilts, this functional has derivative

4 \int f^{4} g = P_{f} (4 f^{3} g)

. At a bounded baseline density its canonical gradient, relative to these regular paths, is

4 (f^{3} - \int f^{4})

. A common local bound on nearby densities suffices to obtain the same derivative along arbitrary differentiable-in-quadratic-mean paths in that bounded neighborhood. A bounded baseline alone is insufficient, because of the spike obstruction to density-power differentiability.

Spike obstruction to density-power differentiability

A perturbation can have negligible Hellinger distance but a large integral of its density's fourth power. At the uniform density on

[0, 1]

, let

b (s) = 6 s (1 - s)

[0, 1]

and zero elsewhere, and let

f_{t} (u) = 1 - t^{4} + t^{- 2} b (u / t^{6})

. Then

\int f_{t} = 1

and

\int (f_{t} - 1)^{2} \leq 2 t^{4} = o (t^{2})

, so the score function is zero. However

\int f_{t}^{4} \geq 72/ (35 t^{2})

. Thus the density fourth-power functional is not continuous along this differentiable-in-quadratic-mean path, despite a bounded baseline and individually bounded nearby densities.

Pathwise differentiability of a statistical functional ( $D ψ_{P} (g) = P (φ g)$ )

A statistical functional is pathwise differentiable relative to chosen statistical paths if its derivative along every path depends only on that path's score function and defines a bounded linear functional on their statistical tangent space. The Riesz representation theorem expresses this derivative as an L2 inner product with a unique element of the statistical tangent space, the canonical gradient. A path family and the associated derivative remainder conditions must both be specified; a formal derivative along one convenient family does not establish differentiability along all paths.

Influence-function representer ( $φ \in L_{0}^{2} (P), D ψ_{P} (g) = P (φ g)$ )

An influence-function representer is a centered square-integrable function representing derivatives of a statistical functional along all admissible score functions. Representers may differ by a function orthogonal to the statistical tangent space. This pathwise definition is distinct from defining an influence function solely by point-mass contamination paths.

Canonical gradient ( $φ_{eff} = Π_{T} φ$ )

The canonical gradient is the influence-function representer in the statistical tangent space

T

. It equals the orthogonal projection onto

T

of any representer. Every other representer differs from it by an element of

T^{⊥}

, so the Pythagorean theorem in an inner-product space gives it minimum variance. For

ψ (P) = P a

in an unrestricted density model with bounded

a

, it is

a - P a

Statistical estimation ( $θ$ )

Statistical estimation uses observed data to infer numerical properties of a population or model. A point estimator gives a value, while a confidence interval describes uncertainty with a specified coverage property. Maximum likelihood estimation is one approach; bias, variance, identifiability and the sampling or selection mechanism all affect an estimate's interpretation.

Estimation error ( $θ - θ$ )

An estimation error is the difference between an estimator and the statistical parameter it estimates. It is a random variable before observing the sample. Its mean is the bias of the estimator, and its mean square is the sum of the variance and squared bias. Standardizing an estimation error by its standard error often yields an approximate standard normal distribution, allowing construction of a Wald confidence interval.

Statistical estimate ( $θ (x)$ )

A statistical estimate is the realized numerical value of an estimator after observing a statistical sample. Before observation, the estimator is a random variable or random vector; its realized value is the estimate. Both are commonly written

θ

, so context distinguishes them.

Ratio estimator ( $θ = U_{n} / V_{n}$ )

A ratio estimator divides one statistic by another, for example an estimated joint moment by an estimated marginal probability. Its definition must specify what happens when the denominator is zero. A denominator bounded away from zero permits direct error bounds through

U_{n} - θ V_{n}

Absolute error bound for a ratio estimator with a controlled denominator ( $E ∣ U_{n} / V_{n} - θ ∣ \leq δ^{- 1} (∣ E G_{n} ∣ + Var G_{n}) + r_{n}$ )

Set

G_{n} = U_{n} - θ V_{n}

, and define the estimator on zero-denominator events as part of the model. On

V_{n} > δ > 0

, its error is at most

∣ G_{n} ∣/ δ

. Centering

G_{n}

and applying the Cauchy-Schwarz inequality gives the displayed bound, where

r_{n}

is the expected error on

V_{n} \leq δ

. This controls a ratio estimator without falsely assuming that its numerator and denominator are independent.

Probability sampling

Probability sampling chooses observational units by a specified random mechanism with known inclusion probabilities. This separates design-based representativeness from the convenience of obtaining particular observations. Simple random sampling, stratified sampling and cluster sampling are different designs with different precision and cost properties.

Sampling with replacement

Sampling with replacement returns each selected label to the available population before the next draw. Independent uniform draws from a fixed population are independent and identically distributed; a label can recur. Replacement by itself does not imply uniformity or independence.

Inclusion probability ( $π_{i} = P (i \in A)$ )

An inclusion probability is the probability that a particular unit belongs to the selected sample. For sample size

n

in simple random sampling from

N

labels it is

n / N

. It differs from the selection probability at a particular draw position.

Unequal probability sampling

Unequal probability sampling assigns different inclusion probabilities to different population units. Estimating an unweighted population summary generally requires accounting for these differences in the estimator.

Finite population mean ( $\overset{ˉ}{Y} = N^{- 1} \sum_{i = 1}^{N} y_{i}$ )

The finite population mean for

N \geq 1

is the arithmetic mean of all fixed values in the target population. Under simple random sampling, the sample mean is an unbiased estimator of it.

Finite population variance ( $S^{2} = (N - 1)^{- 1} \sum_{i} (y_{i} - \overset{ˉ}{Y})^{2}$ )

For

N > 1

, the finite population variance measures spread about the finite population mean. Survey-sampling formulas commonly use denominator

N - 1

; the denominator-

N

version is

σ_{N}^{2} = (N - 1) S^{2} / N

. The normalization must be specified when using a finite population correction.

Sampling without replacement

Sampling without replacement selects distinct labelled units: after a unit is selected it is unavailable for the remaining draws. Distinct labels can carry equal numerical values, so repeated values do not necessarily imply a repeated unit. The rule alone does not specify selection probabilities; a procedure favoring particular units is still sampling without replacement but need not be simple random sampling.

In uniform sequential sampling without replacement, the next label is chosen uniformly among those remaining. For a population of

N

labels, any ordered sequence of

n

distinct labels has probability

1/ [N (N - 1) \dots (N - n + 1)]

. Each size-

n

subset has

n!

possible orders, so its probability is

(n N)^{- 1}

. This gives the fixed-size simple random sampling design. Equivalently, take the first

n

labels of a uniformly random permutation. Restricting a uniformly random permutation to a specified label set preserves uniformity; biased modulo mappings should not replace uniform selection.

The draw positions have identical marginal probability distributions, but exhibit statistical dependence. If the first label is

i

, the conditional probability that the second label is

i

is zero, whereas its marginal probability is

1/ N

. Thus these draws are not independent and identically distributed when

N > 1

and

n > 1

. For a marked subset of labels, counting size-

n

subsets with binomial coefficients gives the hypergeometric distribution. The resulting negative covariance explains the finite population correction for both counts and sample means.

Covariance of draws without replacement ( $Cov (Z_{1}, Z_{2}) = - σ_{N}^{2} / (N - 1)$ )

Draw two distinct positions uniformly from a fixed population

z_{1}, \dots, z_{N}

with

N > 1

. Let

μ = N^{- 1} \sum_{i} z_{i}

and

σ_{N}^{2} = N^{- 1} \sum_{i} (z_{i} - μ)^{2}

. Each draw has expected value

μ

, while

E [Z_{1} Z_{2}] = \frac{( \sum _{i} z _{i} ) ^{2} - \sum _{i} z _{i}^{2}}{N ( N - 1 )} = μ^{2} - \frac{σ _{N}^{2}}{N - 1} .

Subtracting the product of the expected values proves the covariance formula and explains the finite population correction. If the sampled values are coin biases and the actual tosses are conditionally independent given those biases, the law of total covariance gives the same covariance for their head indicator random variables.

Marginal and pair probabilities in sampling without replacement ( $P (B_{i}) = K / N, P (B_{i} \cap B_{j}) = K (K - 1) / [N (N - 1)]$ )

Take a uniformly random permutation of

N > 1

distinguishable objects, of which

K

are marked. Let

B_{i}

mean that position

i

is marked. Every object is equally likely at a specified position, so

P (B_{i}) = K / N

. For distinct positions

i, j

, their ordered pair is uniform among

N (N - 1)

distinct-object pairs, of which

K (K - 1)

are marked pairs. This proves the second formula. For

K > 0

, the conditional probability is

P (B_{i} ∣ B_{j}) = (K - 1) / (N - 1)

For the count

M_{m}

of marked objects among the first

m

positions, indicator random variables and linearity of expectation give

E [M_{m}] = m K / N

. These draws are exchangeable but generally dependent; equal marginal probabilities do not imply independence.

First return to equal counts in sampling without replacement ( $P (T = 2 m) = 2 C_{m - 1} (g)_{m} (r)_{m} / (g + r)_{2 m}$ )

Draw balls sequentially without replacement from

g

green and

r

red balls. Let

T

be the first positive draw count at which the numbers of drawn green and red balls agree. For

1 \leq m \leq min (g, r)

P (T = 2 m) = 2 C_{m - 1} \frac{( g ) _{m} ( r ) _{m}}{( g + r ) _{2 m}}, C_{k} = \frac{1}{k + 1} (k 2 k),

where

(q)_{j} = q (q - 1) \dots (q - j + 1)

denotes the falling factorial and

C_{k}

is a Catalan number. Odd first return times are impossible because equality requires an even total number of draws.

To count the successful colour orders, encode green by a step

+ 1

and red by a step

- 1

. If the first step is positive, a first return time

2 m

means that the partial sums remain strictly positive until the last step. Delete the first positive step and the last negative step, and shift the remaining heights down by one. The result is a Dyck path of length

2 m - 2

, of which there are

C_{m - 1}

. Interchanging the colours doubles this count. Every such colour order has the same probability under sampling without replacement, namely

(g)_{m} (r)_{m} / (g + r)_{2 m}

, proving the formula. In particular, the numbers of successful colour orders at lengths two, four and six are two, two and four; balanced orders with an earlier return must not be counted.

First marked item in a random permutation ( $Pr (K = k) = (m - 1 N - k) / (m N)$ )

When

m \geq 1

items among

N

are marked, inspect a uniformly random permutation until the first marked item. For its first position to be

k

, the other

m - 1

marked positions must lie among the last

N - k

slots. This yields the displayed law with support

1 \leq k \leq N - m + 1

. It is the first-success form of sampling without replacement. For one marked item the position is uniform, while its conditional success chance after

r

failures increases to

1/ (N - r)

Cluster sampling

Cluster sampling chooses groups of observational units, such as production days, rather than dispersed individual units. Testing every unit in a selected group is a one-stage cluster sample. Positive within-cluster intraclass correlation reduces the information supplied by a fixed number of sampled units, although collecting clustered observations can be cheaper.

Design effect ( $D = 1 + (m - 1) ρ$ )

The design effect is a sampling variance under a complex design divided by the corresponding simple-random-sampling variance. For independent equal-size clusters of size

m

with exchangeable intraclass correlation

ρ

, a common mean-estimation approximation is

1 + (m - 1) ρ

. It expresses the loss of effective sample size from positive correlation; finite-population and unequal-cluster-size designs require their own calculation.

Stratified sampling

Stratified sampling partitions a population into strata and samples separately within each. It guarantees specified representation and can improve precision when the strata describe population variation. Equal sampling fractions give equal unit inclusion probabilities; unequal fractions require appropriate weighting for population summaries.

Optimal cost-constrained stratified sampling allocation ( $x_{i}^{*} = \frac{b}{A} v_{i} / a_{i}, A = \sum_{i} a_{i} v_{i}$ )

Suppose the variance is

\sum_{i} v_{i} / x_{i}

and the sampling cost is

\sum_{i} a_{i} x_{i} \leq b

, with positive costs, variance coefficients and budget. The Cauchy-Schwarz inequality gives

A^{2} \leq (\sum_{i} v_{i} / x_{i}) (\sum_{i} a_{i} x_{i}) \leq b \sum_{i} v_{i} / x_{i}

. Equality forces

x_{i}

proportional to

v_{i} / a_{i}

and a tight budget, proving the displayed unique optimum. The minimal variance is

A^{2} / b

. For the optimization Lagrangian

f - λ (\sum_{i} a_{i} x_{i} - b)

, the multiplier and budget derivative both equal

- A^{2} / b^{2}

. Therefore a small resource change

δ b

changes the optimum by

λ δ b + O ((δ b)^{2})

. This is a continuous allocation; integer sample-size restrictions define a separate optimization problem.

Simple random sampling ( $P (A = a) = (n N)^{- 1}$ )

Here simple random sampling means a random unordered subset

A

of exactly

n

distinct labels from

{1, \dots, N}

, with every size-

n

subset equally likely. The sample size is fixed in advance,

1 \leq n \leq N

, and sampling without replacement prevents duplicate units. This is SRSWOR; some texts also use "simple random sampling" for independent uniform draws with replacement, which is a different design.^[ref]

Every label has inclusion probability

n / N

, but equal inclusion probabilities alone are insufficient. For example, from six labels choose either

{1, 2, 3}

{4, 5, 6}

with probability

1/2

. Each label is included with probability

1/2

, yet eighteen of the twenty size-three subsets are impossible: this is a two-group cluster sampling design. Unequal probability sampling deliberately changes inclusion probabilities, often to improve precision for a particular estimand, and generally requires an appropriately weighted estimator rather than the plain sample mean.

Consider the original mathematical population with labelled values

(2, 4, 6, 8, 10, 12)

, and let

n = 2

. The fifteen equiprobable subsets, followed by their sample means, are

first value 2 46810 {2, 4} : 3 {4, 6} : 5 {6, 8} : 7 {8, 10} : 9 {10, 12} : 11 {2, 6} : 4 {4, 8} : 6 {6, 10} : 8 {8, 12} : 10 {2, 8} : 5 {4, 10} : 7 {6, 12} : 9 {2, 10} : 6 {4, 12} : 8 {2, 12} : 7

Their sample means sum to

105

, giving expected value

105/15 = 7

. Their squared deviations from

7

sum to

70

, giving variance

70/15 = 14/3

. This is an enumeration over hypothetical selections from six stipulated numbers, not measured survey data.

Figure 1.
All fifteen size-two subsets and all thirty-six ordered replacement outcomes from six labelled values
. **Details**.

With independent uniform sampling with replacement, the

6^{2} = 36

ordered outcomes each have probability

1/36

. The two orders

(2, 4)

and

(4, 2)

are distinct outcomes, and

(2, 2)

is now possible. Forgetting order does not make the resulting unordered multisets equiprobable:

{2, 4}

has probability

2/36

but

{2, 2}

has probability

1/36

. The thirty-six sample means still have expected value

7

, but their squared deviations sum to

210

, giving variance

210/36 = 35/6

. Avoiding duplicate labels improves precision for this fixed population.

Sample mean under simple random sampling ( $Var (\overset{y}{ˉ}_{A}) = (1 - n / N) S^{2} / n$ )

Fix

N > 1

real population values

y_{1}, \dots, y_{N}

. Define the finite population mean and two normalizations of the finite population variance by

\overset{ˉ}{Y} = \frac{1}{N} \sum_{i = 1}^{N} y_{i}, S^{2} = \frac{1}{N - 1} \sum_{i = 1}^{N} (y_{i} - \overset{ˉ}{Y})^{2}, σ_{N}^{2} = \frac{1}{N} \sum_{i = 1}^{N} (y_{i} - \overset{ˉ}{Y})^{2} = \frac{N - 1}{N} S^{2} .

The values are fixed; only the selected subset

A

is random. For its sample mean

\overset{y}{ˉ}_{A} = n^{- 1} \sum_{i \in A} y_{i}

, the exact design expectation and variance are

E [\overset{y}{ˉ}_{A}] = \overset{ˉ}{Y}, Var (\overset{y}{ˉ}_{A}) = (1 - \frac{n}{N}) \frac{S ^{2}}{n} = \frac{N - n}{N - 1} \frac{σ _{N}^{2}}{n} .

The sample mean is an unbiased estimator of the finite population mean. At

n = N

, every unit is observed and

\overset{y}{ˉ}_{A} = \overset{ˉ}{Y}

exactly, so the variance is zero. The

N = 1

case is deterministic and is handled directly, since

S^{2}

is undefined then.^[ref]

The finite population correction depends on the baseline. With the

N - 1

normalization it is

1 - n / N

multiplying

S^{2} / n

. With the

N

normalization it is

(N - n) / (N - 1)

multiplying

σ_{N}^{2} / n

, the exact variance for independent uniform sampling with replacement. The standard error uses the square root of the relevant variance factor. Mixing the two normalizations gives a wrong answer even at

n = 1

, when replacement cannot matter. For the six-value example,

\overset{ˉ}{Y} = 7

\sum_{i} (y_{i} - 7)^{2} = 70

S^{2} = 14

and

σ_{N}^{2} = 35/3

, reproducing

14/3

without replacement and

35/6

with replacement.

A short proof exposes the statistical dependence. Let

X_{r}

be the value at position

r

in uniform sequential sampling without replacement, so each

X_{r}

has expected value

\overset{ˉ}{Y}

and variance

σ_{N}^{2}

. Write

d_{i} = y_{i} - \overset{ˉ}{Y}

. Every ordered pair of distinct labels is equally likely, and

\sum_{i} d_{i} = 0

, hence for

r \neq = s

Cov (X_{r}, X_{s}) = \frac{\sum _{i \neq = j} d _{i} d _{j}}{N ( N - 1 )} = \frac{( \sum _{i} d _{i} ) ^{2} - \sum _{i} d _{i}^{2}}{N ( N - 1 )} = - \frac{σ _{N}^{2}}{N - 1} = - \frac{S ^{2}}{N} .

Linearity of expectation proves unbiasedness without requiring independence. The variance of a sum includes all

n (n - 1)

ordered covariance terms, giving

Var (\frac{1}{n} \sum_{r = 1}^{n} X_{r}) = \frac{n σ _{N}^{2} - n ( n - 1 ) σ _{N}^{2} / ( N - 1 )}{n ^{2}} .

(4)

For the original population, each pair of distinct draw positions has covariance

- 7/3

. Drawing a high value leaves a lower mean among the remaining values, balancing subsequent draws. Equal marginal probability distributions therefore coexist with negative covariance.

When

2 \leq n \leq N

, the sample variance

s_{A}^{2} = (n - 1)^{- 1} \sum_{i \in A} (y_{i} - \overset{y}{ˉ}_{A})^{2}

is an unbiased estimator of

S^{2}

. To check this independently, use

\sum_{i \in A} (y_{i} - \overset{y}{ˉ}_{A})^{2} = \sum_{i \in A} (y_{i} - \overset{ˉ}{Y})^{2} - n (\overset{y}{ˉ}_{A} - \overset{ˉ}{Y})^{2}

: its expected value is

n σ_{N}^{2} - n Var (\overset{y}{ˉ}_{A}) = (n - 1) S^{2}

. Thus

(1 - n / N) s_{A}^{2} / n

unbiasedly estimates the variance of the sample mean; its square root estimates the standard error, but need not be unbiased.^[ref]

Figure 1.
Exact mean variance and expected distinct-unit coverage as sample size grows in the six-value population
. **Details**.

The diagram compares variance with the expected fraction of distinct population labels observed. Without replacement this fraction is exactly

n / N

; with independent uniform sampling with replacement, a label is missed with probability

(1 - 1/ N)^{n}

, so linearity of expectation gives expected fraction

1 - (1 - 1/ N)^{n}

. At six draws, replacement covers only about

66.5%

of labels on average and retains sample mean variance

35/18

; sampling all six distinct labels covers the population completely and removes selection uncertainty. This population coverage is distinct from confidence interval coverage probability.

The correction matters when the sampling fraction is appreciable. It calibrates uncertainty about this fixed population under the specified selection mechanism. It does not correct unequal selection weights, missing observations or measurement errors; observing every label also does not eliminate uncertainty about a larger unobserved population. Moreover, an exact variance alone does not guarantee the coverage probability of a proposed confidence interval: a small discrete sampling distribution can make a normal approximation inaccurate.

Sampling fraction ( $f = n / N$ )

The sampling fraction is the sample size divided by the finite population size. In simple random sampling, it determines how much the finite population correction reduces the variance relative to

S^{2} / n

Scoring rule ( $S (P, x)$ )

A scoring rule turns a probability distribution forecast and an observed outcome into a number. For example, a forecast of rain with probability

0.7

should express uncertainty more usefully than a bare prediction of rain. A scoring rule lets us compare such forecasts after observations arrive, and its choice determines whether exaggerating confidence pays. A proper scoring rule rewards an honest account of uncertainty in expected value.

There are two sign conventions. A reward

S (P, x)

is maximized; a loss function

L (P, x)

is minimized. The conversion

L = - S

reverses every comparison. Here the linear probability score is a reward, while the binary Brier score and the negative logarithmic scoring rule are losses. Calling all three a "score" without specifying the direction would hide the central difference.

Throughout the binary examples, the random variable

Y

has a Bernoulli distribution with actual success probability

q = 0.7

, and the report

p \in [0, 1]

assigns probability

p

Y = 1

and

1 - p

Y = 0

. The value

q = 0.7

is stipulated by this statistical model; it is not inferred from data or a claim that a forecasting system is calibrated. In practice the true predictive distribution is unknown, and the relevant

q

is the conditional probability of success given the information available when the forecast is issued. Propriety evaluates incentives for reporting that predictive distribution; it does not supply the unknown probabilities.

Linear probability score ( $S (p, x) = x p + (1 - x) (1 - p)$ )

The linear probability score pays the reported probability of whichever binary outcome occurs. With report

p

and observed

y \in {0, 1}

, its reward is

S_{lin} (p, y) = y p + (1 - y) (1 - p) .

It looks sensible: assign more probability to what happens and receive more reward. But the expected value under a Bernoulli distribution with true success probability

q

E_{q} [S_{lin} (p, Y)] = qp + (1 - q) (1 - p) = 1 - q + (2 q - 1) p .

This is an affine function of the report. When

q > 1/2

, the maximum over

[0, 1]

is at

p = 1

; when

q < 1/2

, it is at

p = 0

. At

q = 1/2

, every report ties. Thus the linear probability score encourages reporting certainty about the more likely outcome and is not a proper scoring rule.

For the stipulated

q = 0.7

, the expected value of the reward is

0.3 + 0.4 p

. Honest reporting

p = 0.7

earns

0.58

in expectation, whereas

p = 1

earns

0.7

. The forecaster improves expected reward by replacing a 70% forecast with a 100% forecast. The Brier score and logarithmic scoring rule avoid this incentive, as the expected-score diagram under proper scoring rule shows.

Proper scoring rule ( $E_{Q} S (Q, X) \geq E_{Q} S (P, X)$ )

A proper scoring rule makes reporting the true probability distribution optimal in expected value, for every true probability distribution in the allowed class of forecasts. In the reward convention, the inequality in the title holds for every report

P

and true law

Q

. The expected value is always taken under

Q

, because the report does not change how the outcome is generated. In the loss convention the same definition reads

E_{Q} [L (Q, X)] \leq E_{Q} [L (P, X)] for every admissible P, Q .

This requirement makes a proper scoring rule useful for eliciting probabilities and comparing predictive distributions: deliberately distorting a known forecast cannot improve its expected reward or expected loss. The reward convention and its strict version are set out in Strictly Proper Scoring Rules, Prediction, and Estimation by Tilmann Gneiting and others (Section 1).

For the Bernoulli distribution with

q = 0.7

, the Brier score has expected loss

(p - 0.7)^{2} + 0.21

, and the negative logarithmic scoring rule has expected loss

- 0.7 ln p - 0.3 ln (1 - p)

. Both are minimized at the honest report

p = 0.7

. The linear probability score, by contrast, maximizes its expected reward at

p = 1

. These are statements about averaging over both possible outcomes with their actual probabilities.

Figure 1.
Expected losses of the Brier score and logarithmic scoring rule favor $p = q = 0.7$ ; the linear probability score rewards $p = 1$
. **Details**.

The left panel displays each excess expected loss divided by that rule's expected loss at

p = q

0.21

for the Brier score and approximately

0.610864

for the logarithmic scoring rule. This normalization permits comparison of curve shapes and preserves the minimizing report, since it subtracts a constant and divides by a positive constant independent of

p

. Zero means no excess expected loss. The right panel shows the unmodified linear probability score reward, with its maximizing direction labeled explicitly.

Strictly proper scoring rule

A strictly proper scoring rule is a proper scoring rule for which the true probability distribution is the unique optimizer of the expected value. Equivalently, any different admissible probability distribution gives strictly larger expected loss. For probability density functions, two densities that agree almost everywhere represent the same probability distribution. Ordinary propriety permits ties: the constant reward

S (P, x) = 0

is a proper scoring rule, because every report ties, but it is not a strictly proper scoring rule and elicits no information.

Strict propriety concerns expectation, not guaranteed success on each observation or each realized dataset. With the stipulated Bernoulli distribution

q = 0.7

, suppose the outcome is

Y = 0

. The honest report

p = 0.7

then has Brier score loss

0.49

, while

p = 0.2

has loss

0.04

. The logarithmic loss similarly gives losses

- ln (0.3) \approx 1.203973

and

- ln (0.8) \approx 0.223144

. The honest forecast loses this comparison on that draw, which occurs with probability

0.3

in the stipulated statistical model.

Figure 1.
Individual outcomes can favor a different report even though the expected Brier score and logarithmic loss are uniquely minimized at $p = 0.7$
. **Details**.

The

Y = 0

loss curves favor

p = 0

, and the

Y = 1

loss curves favor

p = 1

. Their weighted average, with weights

0.3

and

0.7

, favors

p = 0.7

. Choosing a report after observing

Y

would use information unavailable at forecast time. Even an average loss on a finite dataset can rank an honest forecast below a competing forecast through outcome variation; strict propriety guarantees the ranking in expected value under the specified information and true predictive distribution.

Brier score ( $L_{B} (p, y) = (p - y)^{2}$ )

The binary Brier score applies squared-error loss to a reported success probability

p

and the observed indicator

y \in {0, 1}

. It is a loss, so smaller is better. Its values lie in

[0, 1]

: predicting the outcome with certainty gives zero, while assigning it zero probability gives one. Unlike the logarithmic scoring rule loss, the binary Brier score remains bounded even for a confidently wrong forecast.

For a Bernoulli distribution with true success probability

q

, averaging the two squared losses gives

E_{q} [(p - Y)^{2}] = q (p - 1)^{2} + (1 - q) p^{2} = (p - q)^{2} + q (1 - q) .

The report controls only

(p - q)^{2}

; the remaining term is the variance of the Bernoulli random variable. Consequently the Brier score is a strictly proper scoring rule. At

q = 0.7

, the expected loss is

(p - 0.7)^{2} + 0.21

, with unique minimum

0.21

p = 0.7

. Reporting

p = 0.2

instead gives expected loss

0.46

, despite its better loss on the particular outcome

Y = 0

above.

For several possible categories, the Brier score sums the squared differences between reported probabilities and observed category indicators. With two categories that sum is

2 (p - y)^{2}

, so its numerical values are twice the binary convention used here, with the same optimizing report. Brier's original ^[ref] averages these category-sum losses over forecasts. Strictly Proper Scoring Rules, Prediction, and Estimation by Tilmann Gneiting and others (Section 3.1, Example 1, and Table 1) gives the category-sum and binary forms as negative rewards; negating them gives the losses described here.

Logarithmic scoring rule ( $S (p, y) = lo g p (y)$ )

The logarithmic scoring rule rewards the natural logarithm of the reported probability mass or probability density at the observed outcome. The title uses the reward convention. For a binary report

p

, its negative is the logarithmic loss:

L_{l o g} (p, y) = - y ln p - (1 - y) ln (1 - p), 0 < p < 1.

For

y = 1

this is

- ln p

; for

y = 0

it is

- ln (1 - p)

. At an endpoint, use these outcome-specific forms: assigning zero probability to the outcome incurs infinite loss, while assigning it probability one gives zero. The natural logarithm therefore makes a confidently wrong binary forecast far more costly than the bounded Brier score does.

Under the stipulated Bernoulli distribution with

q = 0.7

, the expected logarithmic loss is

E_{0.7} [L_{l o g} (p, Y)] = - 0.7 ln p - 0.3 ln (1 - p) .

Its unique minimum is at

p = 0.7

, where the expected loss is approximately

0.610864

. At

p = 0.2

it is approximately

1.193550

. More generally, the excess expected logarithmic loss over truthful reporting is the Kullback-Leibler divergence in the order true law first, reported law second:

E_{q} [L_{l o g} (p, Y)] - E_{q} [L_{l o g} (q, Y)] = q ln \frac{q}{p} + (1 - q) ln \frac{1 - q}{1 - p} = D_{KL} (Bernoulli (q) ∥ Bernoulli (p)) .

For

0 < q < 1

, this Kullback-Leibler divergence is nonnegative and vanishes only at

p = q

, explaining why the logarithmic scoring rule is a strictly proper scoring rule. The same Kullback-Leibler divergence identity applies to probability density functions relative to a common measure when the truthful expected logarithmic loss is finite and the comparison expected value is well defined.^[ref] For sequential forecasts, the existing prequential log score identity explains how logarithmic rewards add over observations.

Prequential log score identity ( $\sum_{t} lo g p (y_{t} ∣ y_{< t}) = lo g p (y_{1 : T})$ )

The probability chain rule makes a sum of sequential predictive log scores equal the log Bayesian model evidence when the forecasts are coherent Bayesian one-step predictions under proper priors. Differences of total scores are therefore log Bayes factors.

Statistical degrees of freedom ( $ν$ )

The number of free dimensions remaining after specified constraints often determines a reference sampling distribution. In a full-rank normal linear model with

n

observations and

p

mean coefficients, the residual space has dimension

n - p

. A regular nested likelihood-ratio test instead uses the difference in free statistical parameter dimensions. Boundary constraints may invalidate this regular calibration.

Statistic ( $T = T (X)$ )

A statistic is a measurable function of observed data whose definition does not involve an unknown parameter. It may be scalar or vector valued. Sufficient statistics summarize all information about a parameter in the conditional-distribution sense.

Complete statistic ( $E_{θ} g (T) = 0 \forall θ ⟹ g (T) = 0 a.s. for all θ$ )

A statistic is complete for a parameter family if the displayed implication holds for every measurable

g

integrable under all parameter values. This property does not itself assert sufficiency; a complete sufficient statistic has both properties. A nonconstant unbiased estimator of a known constant provides a direct obstruction to completeness.

Boundedly complete statistic

Bounded completeness requires the defining implication for a complete statistic only for bounded measurable functions. Completeness implies it, but the converse need not hold. If a sufficient statistic

T

gives a parameter-independent event probability

c

and the conditional probability

S (T)

of that event is nonconstant, then

S - c

is a bounded zero-mean obstruction to bounded completeness.

Meta-analysis

A meta-analysis combines statistical evidence from several studies addressing a common estimand. It may assume a shared effect or use a random-effects meta-analysis to represent between-study variation. Inverse-variance weighted means combine independent effect estimates when their variances are known or estimated; differences in study validity and effect modifiers require substantive assessment, not just weighting.

Subgroup analysis in meta-analysis

Divide studies by a specified characteristic and estimate an effect in each subgroup, then test the difference using an independent subgroup contrast in meta-analysis or an appropriate interaction model. The test of a between-group difference is distinct from separate tests against no effect within each group. Exploratory splits can be affected by multiple testing and by confounding between study characteristics.

Forest plot

A forest plot shows each study's effect estimate and confidence interval, together with a pooled estimate when appropriate. A vertical null-effect line allows comparison on a common scale. For an odds ratio the null is one and a logarithmic axis treats reciprocal ratios symmetrically; marker areas commonly encode study weights.

Funnel plot

A funnel plot displays study effect estimates against a measure of their precision, often standard error increasing down the vertical axis. Under a common-effect model, imprecise estimates spread more widely around the effect. Asymmetry is evidence of a small-study effect and can be consistent with publication bias, but it has other possible explanations.

Small-study effect

A small-study effect is a systematic relationship between study precision or size and estimated effect. It may result from selective publication, design differences, effect modification or other heterogeneity. A regression of effect on its standard error tests one such relationship; it does not by itself determine which mechanism generated it.

Publication bias

Publication bias arises when the availability of studies depends on the results, such as their significance, magnitude or direction. A meta-analysis of the available studies can then differ systematically from one including all eligible evidence. Funnel plot asymmetry can motivate investigation but does not establish this mechanism; heterogeneity and other small-study effects can also cause asymmetry.

Selection model for publication bias ( $f_{obs} (y) = f (y) π (y) / \int f (v) π (v) d v$ )

A selection model combines a distribution for all study results with a probability

π (y)

that a result becomes available. The observed distribution is reweighted by this probability and renormalized. Assumed or estimated selection mechanisms permit sensitivity analysis of a meta-analysis, but unavailable results generally prevent a fully assumption-free correction.

Odds under at-least-one-positive selection ( $R_{n} = R [1 - β^{n}] / [1 - (1 - α)^{n}]$ )

For

n

studies conditionally independent given the truth of one relationship, the event that at least one is positive has probability

1 - β^{n}

under the alternative and

1 - (1 - α)^{n}

under the null. Its Bayes factor therefore gives the displayed posterior odds. For interior error probabilities these tend back to the prior odds

R

. If power exceeds size, the Bayes factor decreases to one: the quotient of geometric sums is a weighted average of decreasing powers

[β / (1 - α)]^{k}

This result concerns the coarsened selection event. Full observed results instead multiply their individual Bayes factors, and concordant evidence can accumulate without this bound. It is a useful example of information discarded by publication bias.

Trim and fill

Trim and fill estimates missing studies from funnel plot asymmetry by trimming extreme studies to locate a centre and filling in counterparts on the less represented side. The adjusted meta-analysis includes these imputed estimates. Its interpretation depends on a symmetry model for missing evidence; it is not a guarantee that genuine publication bias has been removed.

Independent subgroup contrast in meta-analysis ( $Z = (μ_{1} - μ_{2}) / v_{1} + v_{2}$ )

For independent subgroup estimates

μ_{1}, μ_{2}

with variances

v_{1}, v_{2}

, the null contrast has Wald statistic

Z = (μ_{1} - μ_{2}) / v_{1} + v_{2}

. If evidence overlaps, include minus twice the covariance in the contrast variance. A nonsignificant comparison is not proof of equivalence.

Within-study bias

Systematic displacement of an individual study estimate caused by design, conduct, analysis or selective reporting. Combining estimates can reduce random error without reducing this bias. Bias-adjusted meta-analysis and analyses restricted or stratified by relevant bias domains can assess its impact.

Bias-adjusted meta-analysis

Model study estimates as noisy versions of the target effects plus study-specific biases. External information or sensitivity parameters describe those biases and their uncertainty. Removing an assumed bias without propagating its uncertainty overstates precision; observed study data alone generally do not identify the adjustment.

Fixed-effect meta-analysis ( $μ = \sum_{i} w_{i} y_{i} / \sum_{i} w_{i}$ )

A common-effect synthesis treats study estimates as estimates of one shared effect. For independent approximately unbiased estimates

y_{i}

with variances

v_{i}

, use

μ = \sum_{i} w_{i} y_{i} / \sum_{i} w_{i}

with

w_{i} = v_{i}^{- 1}

. This model does not accommodate additional between-study heterogeneity.

Mantel–Haenszel pooled odds ratio ( $OR_{MH} = \frac{\sum _{i} a _{i} d _{i} / n _{i}}{\sum _{i} b _{i} c _{i} / n _{i}}$ )

For independent two-by-two tables with counts

a_{i}, b_{i}, c_{i}, d_{i}

and total

n_{i}

, the Mantel–Haenszel pooled odds ratio is the displayed ratio of sums. When all

b_{i} c_{i} > 0

, it is also the weighted arithmetic mean of study odds ratios with weights

b_{i} c_{i} / n_{i}

. These differ from inverse estimated variance weights for log odds ratios. Both constructions can represent a fixed-effect meta-analysis, but they should not be silently interchanged when interpreting a supplied plot.

Leave-one-study-out influence analysis ( $μ - μ_{(- i)}$ )

Refit a meta-analysis after deleting each study, comparing fitted effects, uncertainty and heterogeneity to the full fit. For fixed weights

w_{i} > 0

, total weight

W

and weighted effect

μ

, direct subtraction of the two weighted means gives

μ - μ_{(- i)} = w_{i} (y_{i} - μ) / (W - w_{i})

. Re-estimating heterogeneity additionally measures influence through the weights. This is a sensitivity diagnostic, not an automatic rule for removing discordant studies.

Random-effects meta-analysis ( $y_{i} \sim N (μ, v_{i} + τ^{2})$ )

For independent study estimates

y_{i}

with within-study variances

v_{i}

, a usual approximate model is

y_{i} ∣ δ_{i} \sim N (δ_{i}, v_{i})

and

δ_{i} \sim N (μ, τ^{2})

. Its marginal distribution is

N (μ, v_{i} + τ^{2})

, so for fixed heterogeneity

τ^{2}

the inverse-variance weighted mean uses weights

(v_{i} + τ^{2})^{- 1}

. Here

μ

describes the mean effect across comparable studies;

τ^{2}

represents between-study variation. Estimation and uncertainty for heterogeneity are essential, especially with few studies.

Student t random-effect model ( $β_{j} ∣ μ, ψ \sim μ + ψ t_{ν}$ )

A Student t random-effect model assigns Student's t-distributions to exchangeable study effects, allowing heavier tails than a normal distribution hierarchy. An equivalent Gaussian scale mixture is

λ_{j} \sim χ_{ν}^{2}

β_{j} ∣ λ_{j}, μ, ψ \sim N (μ, ν ψ^{2} / λ_{j})

, with independent latent draws. For

ν > 2

the variance is

ν ψ^{2} / (ν - 2)

, so

ψ

is a scale rather than a standard deviation. Small latent precisions weaken shrinkage for atypical studies while retaining partial pooling for the rest.

Between-study heterogeneity ( $τ^{2}$ )

Variation in the underlying study effects, additional to sampling error. The usual normal random-effects meta-analysis models it by a between-study variance

τ^{2}

. It can reflect effect modification, design differences or within-study bias; it is not automatically a biological treatment difference.

Meta-regression

Meta-regression relates study effect estimates to study-level explanatory variables. A common random-effects meta-analysis extension is

y_{i} = α + β x_{i} + u_{i} + ε_{i}

, where

Var (u_{i}) = τ^{2}

and

Var (ε_{i}) = v_{i}

. For log odds ratios,

e^{β}

is a ratio of underlying odds ratios per unit of

x

. Few studies, correlated design changes and exploratory variable selection limit interpretation; a study-level association does not identify an individual-level causal effect. The interpretation of study-level predictors is also discussed in the Cochrane methods handbook, section 10.11.

Prior calibration for normal random-effect range ( $τ \leq R / (2 z_{1 - ε / (2 (2 J))})$ )

For

J \geq 2

conditionally independent effects

β_{j} \sim N (μ, τ^{2})

, each pair difference has normal distribution

N (0, 2 τ^{2})

. With

m = (2 J)

, an upper bound

A = R / [2 Φ^{- 1} (1 - ε / (2 m))]

τ

ensures, by the union bound, that

P (max_{j} β_{j} - min_{j} β_{j} > R) \leq ε

. Any proper scale prior distribution supported on

(0, A)

preserves this bound after averaging. This is conservative simultaneous calibration, rather than the weaker statement about one selected pair. For log odds ratios, a bound

R

corresponds to a ratio-of-odds-ratios bound

e^{R}

I-squared statistic ( $I^{2}$ )

For

Q > 0

and

m

studies,

I^{2} = max {0, [Q - (m - 1)] / Q}

. Usually reported as a percentage, it describes excess variation relative to the observed variation on this heterogeneity scale. It is not a proportion of studies with different effects or a proportion of an individual clinical outcome.

DerSimonian–Laird estimator ( $τ^{2} = max {0, [Q - (m - 1)] / C}$ )

For

m

studies let

w_{i} = v_{i}^{- 1}

W = \sum_{i} w_{i}

C = W - \sum_{i} w_{i}^{2} / W

, and let

Q

be Cochran's Q statistic. The moment estimate is

τ^{2} = max {0, [Q - (m - 1)] / C}

when

C > 0

. It estimates the variance of underlying study effects, not their sampling variance.

Cochran's Q statistic ( $Q = \sum_{i} w_{i} (y_{i} - μ)^{2}$ )

With fixed inverse-variance weights and weighted mean

μ

Q = \sum_{i} w_{i} (y_{i} - μ)^{2}

. Under a common-effect normal model with

m

independent estimates and known variances it has a chi-squared distribution with

m - 1

degrees of freedom. Estimated variances make this calibration approximate.

Network meta-analysis

A network meta-analysis jointly compares several treatments using a graph of direct randomized controlled trials and indirect treatment comparisons. If effects are measured relative to a reference treatment, consistency means

δ_{ij} = δ_{i 0} - δ_{j 0}

. A connected graph identifies all relative effects under that model. Transitivity in network meta-analysis is the substantive comparability assumption supporting these relations; inconsistency can be assessed when the network has loops.

Indirect treatment comparison ( $δ_{A B} = δ_{A C} - δ_{BC}$ )