Statistical model

A probabilistic graphical model represents factorization and conditional-independence structure using a graph whose vertices are random variables.

Markov random field

A Markov random field is an undirected graphical model whose graph encodes conditional independence by vertex separation.

Clique potential

A clique potential is a nonnegative function of the variables belonging to one clique. Products of clique potentials define the unnormalized density of a Markov random field.

Global Markov property for an undirected graph

The global Markov property says that graph separation of vertex sets

A

and

B

S

implies conditional independence of the corresponding random vectors given the variables at

S

Statistical modelling

Statistical modelling constructs, fits, checks, and interprets a statistical model for observed data.

Functional data analysis

Functional data analysis treats each observation as a function or another infinite-dimensional object. Means, covariance operators, spectral coordinates, and regression operators replace their finite-dimensional vector and matrix counterparts.

Functional principal component analysis

Functional principal component analysis diagonalizes a compact covariance operator. Its eigenfunctions are principal component functions, and projecting a centered observation onto them gives uncorrelated functional principal component scores.

Principal component function

A principal component function is a normalized eigenfunction

ϕ_{k}

of a covariance operator, ordered by decreasing eigenvalue

λ_{k}

Functional principal component score ( $ξ_{k}$ )

The functional principal component score of a centered function

X

along

ϕ_{k}

ξ_{k} = ⟨ X, ϕ_{k} ⟩

. Its variance is the corresponding covariance eigenvalue

λ_{k}

Karhunen–Loève expansion

The Karhunen–Loève expansion writes a centered square-integrable random function as

X = \sum_{k} ξ_{k} ϕ_{k}

in mean square, where the

ϕ_{k}

are principal component functions and the uncorrelated scores satisfy

E ξ_{k}^{2} = λ_{k}

Functional mean test

A functional mean test tests whether the mean of a Hilbert-space-valued random variable is zero. The squared-norm statistic

n ∥ \overline{X} ∥^{2}

has a weighted chi-squared limit under the null, while an FPCA test standardizes and truncates the coordinates.

Covariance-operator distance

A covariance-operator distance compares positive trace-class operators while respecting either their linear embedding in operator space or a chosen factorization geometry.

Hilbert-Schmidt distance between covariance operators ( $d_{L}$ )

The linear distance between covariance operators is

d_{L} (C_{1}, C_{2}) = ∥ C_{1} - C_{2} ∥_{HS}

Square-root distance between covariance operators ( $d_{R}$ )

The square-root distance is

d_{R} (C_{1}, C_{2}) = ∥ C_{1}^{1/2} - C_{2}^{1/2} ∥_{HS}

Procrustes distance between covariance operators ( $d_{P}$ )

For factorizations

C_{i} = L_{i} L_{i}^{*}

, the Procrustes distance is

d_{P} (C_{1}, C_{2}) = in f_{R} ∥ L_{1} - L_{2} R ∥_{HS}

, where

R

ranges over unitary operators on the factor space. Equivalently,

d_{P} (C_{1}, C_{2})^{2} = tr C_{1} + tr C_{2} - 2 tr ((C_{1}^{1/2} C_{2} C_{1}^{1/2})^{1/2}) .

Functional linear model

A functional linear model relates functional or scalar responses to functional predictors through a linear operator.

Function-on-function linear model

A function-on-function linear model has

Y (t) = \int β (t, s) X (s) d s + ε (t)

. If the centered error is independent of

X

, then

E [Y ∣ X]

is the regression operator applied to

X

Cross-covariance operator ( $C_{Y X}$ )

For centered Hilbert-space random variables

X

and

Y

, the cross-covariance operator is

C_{Y X} h = E [⟨ X, h ⟩ Y]

. In the functional linear model

Y = BX + ε

with independent centered error,

C_{Y X} = B C_{X}

Goodness of fit

Goodness of fit describes how closely a fitted statistical model agrees with the observed data in aspects relevant to the model assumptions.

A goodness-of-fit test compares a discrepancy statistic with its distribution under a fitted null model. A small p-value indicates that the observed discrepancy would be unusual if that model were correct.

Contingency table

A contingency table records counts classified by combinations of categorical variables.

Two-by-two contingency table ( $2 \times 2$ )

A two-by-two contingency table cross-classifies observations by two binary variables, producing four cell counts and two pairs of marginal totals.

Mixture model

A mixture model represents a population distribution as a weighted combination of component distributions.

Finite mixture model

A finite mixture model draws each observation from one of finitely many component distributions according to a categorical latent label.

Mixture weight ( $π_{k}$ )

A mixture weight is the probability assigned to one component of a mixture model; all mixture weights are nonnegative and sum to one.

Dirichlet process mixture model

A Dirichlet process mixture model uses a Dirichlet process as a random mixing distribution, allowing the number of occupied mixture components to be inferred from the data.

Latent-variable model

A latent-variable model represents observed data through unobserved random quantities whose marginalization induces the observed distribution.

Statistical learning

Statistical learning constructs predictive rules from data and studies their risk, generalization, and computational fitting.

Validation set

A validation set is held out from parameter fitting and used to choose models or tuning parameters. A separate test set is then needed for an unbiased final evaluation after selection.

Prediction error

Prediction error is the expected loss of a fitted rule on a new observation from the target distribution.

Test error

Test error is the average loss on an independent test set and estimates prediction error when the test set has not influenced model fitting or selection.

Mean squared prediction error ( $MSPE$ )

For a predictor

f

and an independent response

Y^{*}

at covariate

X^{*}

, mean squared prediction error is

E [(Y^{*} - f (X^{*}))^{2}]

Training error

Training error is the average loss evaluated on the same observations used to fit the prediction rule. Flexible fitting usually makes it optimistically biased for prediction error.

Data leakage

Data leakage occurs when information unavailable at prediction time, including information from nominally held-out observations, influences model fitting or selection. It makes estimated out-of-sample performance optimistically biased.

Dataset shift

Dataset shift means that the joint distribution of predictors and responses differs between training and deployment data.

Classification in statistical learning

Classification predicts a discrete label from observed covariates.

Decision boundary

A decision boundary is the subset of covariate space across which a classifier changes its predicted class.

Majority vote

A majority-vote ensemble predicts the class selected by more constituent classifiers than any other class.

Confusion matrix

A confusion matrix cross-tabulates true classes against predicted classes. Its diagonal entries count correct predictions and its off-diagonal entries describe the kinds of classification errors.

Classification and regression tree

A classification and regression tree recursively splits covariate space into axis-aligned regions and assigns a constant prediction to each terminal region.

Recursive partitioning

Recursive partitioning repeatedly selects a split of one current region, then applies the same splitting rule independently to the resulting child regions.

Gini impurity

For class proportions

p_{1}, \dots, p_{K}

in a tree node, Gini impurity is

1 - \sum_{k} p_{k}^{2}

. For two classes it is twice

p (1 - p)

; omitting the constant factor does not change the selected split.

Random forest

A random forest aggregates deeply grown decision trees fitted to bootstrap samples, while restricting each split to a random subset of predictor coordinates. Classification uses a majority vote over the trees.

Out-of-bag error

The out-of-bag error predicts each training observation using only trees whose bootstrap samples omitted it, then averages the resulting losses.

Conditional class probability ( $p_{k} (x)$ )

The conditional class probability for class

k

p_{k} (x) = P (Y = k ∣ X = x) .

A Bayes classifier selects a class maximizing this probability.

Risk consistency

A sequence of classifiers

h_{n}

is risk-consistent when its classification risk converges to the Bayes risk as the training sample size tends to infinity.

K-nearest neighbors algorithm

 2  0

The K-nearest neighbors algorithm predicts from the labels or responses of the

K

training observations closest to the query under a chosen distance.

L-nearest-neighbour classifier ( $L$ -NN)

The L-nearest-neighbour classifier estimates each class probability by its empirical frequency among the

L

nearest training covariates and predicts a class of greatest estimated probability.

One-nearest-neighbour classification ( $1$ -NN)

One-nearest-neighbour classification assigns a query the label of its single nearest training observation. Under local regularity, its limiting conditional error is

1 - \sum_{k} p_{k} (x)^{2}

Plug-in classifier excess-risk bound

For binary classification with posterior estimate

p_{1}

, the plug-in classifier satisfies

R (C) - R (C^{*}) \leq 2 E ∣ p_{1} (X) - p_{1} (X) ∣.

Support vector machine

A support vector machine chooses a separating decision function by trading a wide geometric margin against hinge-loss violations.

Separating hyperplane

For signed observations

(x_{i}, y_{i})

, an affine hyperplane with score

f (x) = α + x^{T} β

is separating when

y_{i} f (x_{i}) > 0

for every observation.

Slack variables of a support vector machine ( $ξ_{i}$ )

The slack variables satisfy

ξ_{i} \geq max {0, 1 - y_{i} f (x_{i})}

and quantify margin violations.

Support-vector-machine decision boundary

For a linear score

f (x) = α + x^{T} β

, the decision boundary is

f (x) = 0

Support-vector-machine margin

For a linear support vector machine, the geometric margin lies between

f (x) = - 1

and

f (x) = 1

and has width

2/ ∥ β ∥

Support-vector-machine margin boundaries

The two margin boundaries of a linear support vector machine are the parallel hyperplanes

f (x) = 1

and

f (x) = - 1

Support vector

A support vector is a training observation on or inside the fitted margin; it has a nonzero dual coefficient and can affect the fitted boundary.

Perceptron

The perceptron is an online linear classifier that adds a misclassified observation's signed feature vector to its current parameter vector.

Cross-validation

Cross-validation estimates predictive performance by repeatedly fitting on one part of the data and evaluating on held-out observations.

Leave-one-out cross-validation

Leave-one-out cross-validation performs one fit with each observation held out in turn and averages the held-out losses.

Leave-one-out residual identity for a linear smoother

If a linear smoother has fitted vector

Y = H Y

, its residual after fitting without observation

i

Y_{i} - Y_{- i, i} = \frac{Y _{i} - Y _{i}}{1 - H _{ii}} .

The identity follows from a block matrix inverse or a rank-one inverse update.

K-fold cross-validation

K-fold cross-validation partitions observations into

K

folds, trains

K

times while holding out one fold at a time, and combines the held-out losses.

Neural network

 2  0

A neural network composes affine maps with nonlinear activation functions and learns their weights from data.

Feedforward neural network

A feedforward neural network composes layers without directed cycles, passing each layer's activations only to later layers.

Rectified linear unit

The rectified linear unit activation is

ReLU (x) = max (x, 0)

Gaussian error linear unit

The Gaussian error linear unit activation is

GELU (x) = x Φ (x)

, where

Φ

is the standard normal distribution function.

Softmax function

For logits

z_{1}, \dots, z_{K}

, the softmax probabilities are

p_{k} = e^{z_{k}} / \sum_{j} e^{z_{j}}

Categorical cross-entropy loss

For one-hot label

y

and predicted class probabilities

p

, categorical cross-entropy is

- \sum_{k} y_{k} lo g p_{k}

, the negative categorical log-likelihood.

Overfitting

 2  0

Overfitting occurs when further adaptation improves training performance while degrading performance on new data.

Regularization

Regularization restricts or penalizes model fitting to improve generalization.

Early stopping

Early stopping halts iterative fitting near the best validation performance, limiting adaptation to training noise.

Unbiased estimator

An estimator

T

of a parameter

θ

is unbiased when

E_{θ} T = θ

for every parameter value in the model.

Bias of an estimator ( $Bias_{θ} (θ)$ )

The bias of an estimator is

Bias_{θ} (θ) = E_{θ} θ - θ .

Variance of an estimator ( $Var_{θ} (θ)$ )

The variance of an estimator is the variance of its sampling distribution under the parameter value

θ

Asymptotic variance

The asymptotic variance is the variance appearing in the limiting distribution of a suitably rescaled estimator, commonly

n (θ_{n} - θ)

Bias-variance tradeoff

The bias-variance tradeoff describes how increasing model flexibility commonly decreases systematic approximation bias while increasing sampling variance, or conversely how stronger smoothing decreases variance while increasing bias.

Heteroscedastic

Observations or errors are heteroscedastic when their variances are not all equal.

Akaike information criterion

For a model with

k

fitted parameters and maximized likelihood

L (θ)

, the Akaike information criterion is

AIC = - 2 lo g L (θ) + 2 k .

Mallows Cp

For a normal linear model with known error variance

σ^{2}

, the scaled Akaike criterion differs by a model-independent constant from

C_{p} = ∥ Y - μ ∥^{2} + 2 p σ^{2} .

Bias-variance decomposition for linear prediction

Y, Y^{*}

are independent

N (μ, σ^{2} I_{n})

vectors and

H

is a rank-

p

orthogonal projection, then

E ∥ H Y - Y^{*} ∥^{2} = ∥ (I - H) μ ∥^{2} + (n + p) σ^{2} .

The first term is squared approximation bias, while

p σ^{2}

is fitted-model variance and

n σ^{2}

is irreducible new-response noise.

Unbiased prediction-error identity for ordinary least squares

H

is the rank-

p

orthogonal projection onto a normal linear model's column space, then

E ∥ (I - H) Y ∥^{2} = ∥ (I - H) μ ∥^{2} + (n - p) σ^{2} .

Adding

2 p σ^{2}

gives

∥ (I - H) μ ∥^{2} + (n + p) σ^{2}

, so Mallows'

C_{p}

is unbiased for independent-copy prediction error even when the projection model is misspecified.

Shape parameter

A shape parameter changes the form of a probability distribution without merely translating it or rescaling its random variable.

Exponential family

A density

h (y) exp {θT (y) - A (θ)}

has sufficient statistic

T

and cumulant function

A

Natural parameter of an exponential family

In the canonical representation

f_{θ} (y) = h (y) exp {θT (y) - A (θ)},

θ

is the natural parameter and

T

is the natural statistic.

Cumulant function of an exponential family

The cumulant function, or log-partition function, is the normalizing term

A (θ) = lo g \int h (y) e^{θT (y)} d y,

with a sum in the discrete case.

Exponential-family derivative identities

Differentiating the normalization identity gives

A^{'} (θ) = E_{θ} T, A^{''} (θ) = var_{θ} T .

Mean parameter of an exponential family ( $μ$ )

The mean parameter corresponding to the natural parameter

θ

μ (θ) = E_{θ} [T (Y)] = A^{'} (θ) .

Inverse Gaussian distribution

For an expected value parameter

μ > 0

and a shape parameter

λ > 0

, the inverse Gaussian distribution has probability density function

f (x; μ, λ) = (\frac{λ}{2 π x ^{3}})^{1/2} exp (- \frac{λ ( x - μ ) ^{2}}{2 μ ^{2} x}), x > 0.

Its mean is

μ

and its variance is

μ^{3} / λ

Exponential-family deviance ( $D (θ_{1}, θ_{2})$ )

The deviance from

θ_{1}

θ_{2}

is twice the Kullback-Leibler divergence:

D (θ_{1}, θ_{2}) = 2 E_{θ_{1}} [lo g \frac{f _{θ_{1}} ( Y )}{f _{θ_{2}} ( Y )}] .

Negative binomial exponential family

For fixed positive integer

r

, the failures-before-the-

r

th-success distribution is a one-parameter exponential family with

θ = lo g (1 - p) < 0, T (x) = x, A (θ) = - r lo g (1 - e^{θ}),

and carrier

h (x) = (x x + r - 1)

Exponential dispersion family

With dispersion

ϕ

and known weight

w

, an exponential dispersion density has exponent

\frac{w}{ϕ} {y θ - b (θ)} + c (y, ϕ / w) .

Its mean is

b^{'} (θ)

and its variance is

(ϕ / w) b^{''} (θ)

Dispersion parameter ( $ϕ$ )

The dispersion parameter scales the variance of an exponential dispersion family while leaving its mean function unchanged.

Overdispersion

Overdispersion occurs when observed conditional variation exceeds that prescribed by a statistical model. For count data it commonly means variance greater than the mean assumed by a Poisson model.

Variance function ( $V (μ)$ )

The variance function writes an exponential dispersion family's variance as

Var (Y) = ϕ V (μ) / w

Generalized linear model

A generalized linear model takes independent responses from an exponential dispersion family and relates their means to linear predictors by

g (μ_{i}) = x_{i}^{T} β

. The response distribution, systematic component, link, dispersion, and known observation weights together specify the model.

Linear predictor ( $η = Xβ$ )

The linear predictor of a generalized linear model is the linear combination

η_{i} = x_{i}^{T} β

of the explanatory variables.

Link function

A link function connects the response mean to the linear predictor through

g (μ_{i}) = η_{i}

Logarithmic link function ( $g (μ) = lo g μ$ )