Statistical learning

Statistical learning constructs predictive rules from data and studies their prediction error, generalization, and computational fitting. In supervised learning, training data contain inputs $X$ and a response variable $Y$: regression analysis predicts a numerical response, while classification in statistical learning predicts a label. Unsupervised learning instead seeks structure without supplied responses, as in cluster analysis and principal component analysis. These tasks connect learning to probability and mathematical optimization. ^[ref]

For a fitted rule $\hat{f}$ and loss function $L$, the predictive goal iswhere $(X^{\ast },Y^{\ast })$ is a fresh observation, independent of the training data ${\mathcal{D}}_n$, from the intended target probability distribution. This prediction error fixes the fitted rule and averages over future observations; averaging again over training data gives the learning procedure's average performance. Its practical beauty is that a rule compresses past observations into useful future predictions. Its central difficulty is that excellent training error need not imply good generalization: cross-validation supports model selection, and a reserved test set assesses the resulting choice.

Supervised learning fits a prediction rule from training data containing both inputs and associated target responses. Regression analysis predicts numerical responses; classification in statistical learning predicts class labels. A loss function specifies which prediction errors matter, and generalization concerns performance beyond the observations used to fit the rule.

Generalization is the ability of a fitted prediction rule to perform well on new observations under a specified target probability distribution. It is assessed through prediction error rather than training error. The target distribution and use of data are part of the question: dataset shift can change performance, and data leakage can make an apparent assessment misleading.

Learning structure from data without supplied target labels. Examples include extracting principal components, clustering and learning representations that efficiently reconstruct inputs.

Construct low-dimensional coordinates representing a dissimilarity matrix. Different methods preserve different features: classical multidimensional scaling approximates a centered Gram matrix, while other methods minimize discrepancies between fitted and supplied distances or preserve distance ordering. The method and dimension determine what geometric claims the resulting display supports.

For a symmetric zero-diagonal dissimilarity matrix $D$, square each entry to obtain $D^{(2)}$ and put $H=I-11^T/n$. Take the eigenvectors of $G=-H{D}^{(2)}H/2$ corresponding to its largest positive eigenvalues. Multiplying them by square roots of these eigenvalues gives coordinates. If $D$ contains Euclidean distances, $G$ is a centered Gram matrix, so retaining all positive eigenvalues reconstructs those distances exactly. Retaining fewer approximates the Gram matrix, not generally the sum of squared errors in raw distances. Negative eigenvalues signal failure of an exact Euclidean representation. Coordinates are unchanged in meaning by orthogonal transformations.

For symmetric nonnegative $D$ with zero diagonal, a centered Euclidean realization implies $D_{ij}^2=G_{ii}+G_{jj}-2G_{ij}$ and hence the displayed double-centering identity. Conversely, if the double-centered matrix is positive semidefinite, factor it as $X{X}^T$. Its reconstructed squared distances equal the original squared distances: their difference has zero double-centering, so is of the form $1a^T+a{1}^T$; the zero diagonal forces $a=0$. The minimum exact embedding dimension is $\mathrm{rank}G$.

Represent inputs accurately using few strongly active coefficients, for example by minimizing $\Vert x-Az{\Vert }^2/2+\lambda {\sum }_i|z_i|$. Learning basis functions on natural images can yield localized oriented filters resembling simple-cell receptive fields. This demonstrates one possible functional account of selectivity, not a proof that cortex implements that exact objective or learning algorithm.

Activity-dependent strengthening associated with coactivation of an input and its postsynaptic output. The idealized update $\Delta w_i\propto y{x}_i$ requires normalization, competition or other constraints to prevent indefinite weight growth.

With $y=w\cdot x$, the squared weight norm $s=|w{|}^2$ satisfies $\tau \dot{s}=2y^2(1-\alpha s^2)$. It moves monotonically toward $s_{\ast }={\alpha }^{-1/2}$ without crossing it. Explicitly $(s-s_{\ast })/(s+s_{\ast })$ equals its initial value times $\exp [-(4\sqrt{\alpha }/\tau ){\int }_0^{t}y(u{)}^{2}du]$. Persistent excitation therefore drives $|w|$ to ${\alpha }^{-1/4}$; without excitation it can remain stationary. This nonlinear normalization avoids the unbounded amplification of unmodified Hebbian learning.

For a linear output $y=w^{T}x$, the second term limits the growth produced by the Hebbian term. On the averaged dynamics with $C=\mathbb{E}[x{x}^T]$, $\tau \dot{w}=Cw-\alpha (w^{T}Cw)w$. For $\alpha >0$, positive-eigenvalue equilibria have norm $1/\sqrt{\alpha }$ and align with eigenvectors of $C$. If inputs are centered, $C$ is their covariance matrix.

At the equilibrium $v_i/\sqrt{\alpha }$, linearization along orthonormal eigenvectors of $C$ gives the displayed rates, with the first equation applying to $j\ne i$. A simple positive largest eigenvalue makes its two normalized eigenvectors locally asymptotically stable; smaller eigenvalues have an unstable direction. Repeated largest eigenvalues give a manifold of equilibria with neutral directions. Generic convergence also needs a nonzero initial component in the largest eigenspace. The squared norm satisfies $\tau d\Vert w{\Vert }^2/dt=2(w^{T}Cw)(1-\alpha \Vert w{\Vert }^2)$.

Cluster analysis groups observations using similarities or dissimilarities without supplied class labels. A distance choice specifies which features and scales matter. Agglomerative hierarchical clustering merges groups recursively, producing partitions at different resolutions rather than one intrinsically determined number of groups.

A cluster in cluster analysis is a nonempty group of observations treated as similar under the chosen representation and dissimilarity. A partition assigns each observation to exactly one cluster; agglomerative hierarchical clustering supplies a nested sequence of such partitions. An algorithmic cluster need not correspond to a distinct probabilistic population.

For a prescribed number of groups, minimize within-cluster squared Euclidean distances. At fixed assignments, differentiating the objective gives the mean as the optimal centroid. At fixed centroids, assigning each point to a nearest centroid minimizes its contribution. Alternating these steps cannot increase the objective but can converge to a local optimum, so multiple starts are useful. Empty clusters and ties need an explicit handling rule. The method favors roughly spherical groups and depends on variable scaling.

The index measures similarity of finite sets by shared presence relative to total presence. For binary profiles it ignores joint absences, unlike the simple matching coefficient. When both sets are empty, define the similarity to be one. The complementary Jaccard distance measures dissimilarity.

The dissimilarity is the number of mismatched presences divided by the number of coordinates present in at least one profile. It differs from normalized Hamming distance because coordinates absent from both profiles do not enter its denominator. Define the distance of two empty sets to be zero.

A dissimilarity matrix records pairwise separation of observations or profiles. Usually it is symmetric, nonnegative and zero on the diagonal. A metric space additionally requires the triangle inequality and separation of distinct profiles. Agglomerative hierarchical clustering can be applied to many dissimilarity matrices that are not Euclidean distance matrices; the claimed geometry must be checked separately.

For binary vectors, this similarity counts agreements on both presences and absences. Its complement is the normalized Hamming distance. Whether joint absences should count as similarity depends on the application; rare-presence data may call for a different coefficient.

The displayed distance is the normalized Hamming distance. Each coordinate mismatch between $x,z$ implies a mismatch between $x,y$ or $y,z$, so summing the indicator inequalities proves the triangle inequality. Nonnegativity, symmetry and separation follow immediately. It is a metric on profiles; on separately labelled individuals with identical profiles it is a pseudometric.

Start with singleton clusters and repeatedly merge the two clusters with the smallest chosen intercluster dissimilarity. Single-linkage clustering uses the smallest cross-pair distance; complete-linkage clustering uses the largest. Merge heights and sensitivity to ties help assess a proposed number of groups.

Ward clustering merges the pair causing the smallest increase in within-cluster sum of squares. Expanding each cluster's squared deviations about the merged mean gives the displayed increase. Its Euclidean sum-of-squares interpretation must not be asserted for arbitrary non-Euclidean dissimilarities.

Average linkage merges the pair with the smallest mean cross-pair dissimilarity. This differs from the minimum used by single-linkage clustering and the maximum used by complete-linkage clustering. The convention here weights individual cross-pairs equally; weighting already formed clusters equally is a different algorithm.

The min and max cross-pair definitions of single-linkage clustering and complete-linkage clustering remain meaningful for a symmetric nonnegative dissimilarity matrix that violates the triangle inequality. Such a calculation gives a valid algorithmic dendrogram, but does not turn the original matrix into Euclidean distances. A triangle-inequality counterexample identifies that source defect without requiring invented replacement data.

A dendrogram displays successive cluster merges, with the height of each merge equal to its intercluster dissimilarity. Cutting at a height gives a partition. Single-linkage clustering measures the closest cross-pair and can form chains, while complete-linkage clustering measures the farthest cross-pair and favors smaller within-cluster diameters.

The farthest cross-pair distance determines the next merge. It favors groups with small diameter and reduces the chaining effect of single-linkage clustering, but can be sensitive to isolated extreme observations. Ties can affect intermediate partitions.

The nearest cross-pair distance determines the next merge. At a threshold, clusters are connected components of the graph joining pairs below that threshold. This handles nonconvex shapes but can join otherwise separate groups through a chain of intermediate observations.

A validation set is held out from parameter fitting and used to choose models or tuning parameters. A separate test set is then needed for an unbiased final evaluation after selection.

A test set is held out from both parameter fitting and model selection and is used to estimate the final selected procedure's prediction error. A validation set influences the choice; a test set assesses that choice after it is fixed. The distinction concerns how observations are used, not their file names.

For a fixed rule and an independent test set of size $m$ from the target probability distribution,This test error is an estimate, with sampling uncertainty; it is not a guarantee for each future prediction. Dataset shift changes what performance the estimate represents.

In the overfitting example, all degree choices are fixed in advance, cross-validation uses only training data, and the selected degree is locked before evaluating 400 separate points. Displaying the other curves afterward explains their behavior; it does not authorize selecting by those scores. Repeatedly inspecting a test set and changing the model makes it part of model selection, introducing data leakage and potentially optimistic assessment. ^[ref]

Prediction error is the expected loss of a fitted rule on a new observation from the target distribution.

Test error is the average loss on an independent test set and estimates prediction error when the test set has not influenced model fitting or selection.

For a predictor $\hat{f}$ and an independent response $Y^{\ast }$ at covariate $X^{\ast }$, mean squared prediction error is $\mathbb{E}[(Y^{\ast }-\hat{f}(X^{\ast }){)}^2]$.

Training error is the average loss function evaluated on the observations used to fit the prediction rule. It is the empirical risk of the fitted rule on its own training data. With squared-error loss, it is the average squared regression residual, measuring fit to observed responses rather than performance on fresh responses. Flexible fitting usually makes it optimistically biased for prediction error, as quantified by prediction optimism.

For a reproducible polynomial regression example, take $x_i=-1+2i/11$, $i=0,\dots ,11$, and the known noiseless target $f(x)=1+x/2+2x^2$. Construct responses $y_i=f(x_i)+e_i$ with fixed disturbancesOrdinary least squares over polynomials of polynomial degree at most 1, 2, and 7 gives training errors $0.518937$, $0.015784$, and $0.009257$. The monotone improvement follows because these candidate families are nested: every smaller-degree fit remains available in the larger family. This argument concerns minimum training error, not generalization, and assumes the fitting mathematical optimization is solved. Degrees 2 and 7 fitted instead to the exact noiseless responses $f(x_i)$ have mathematically zero training error; the script obtains approximately $5.31\times 10^{-31}$ and $7.82\times 10^{-31}$ from rounding. Extra flexibility alone does not force overfitting.

Prediction optimism is the difference between expected squared prediction error on new independent responses at the training covariates and expected training error. Reusing responses to fit and assess a model tends to make training error too favorable. The covariance formula for prediction optimism measures this reuse without assuming a linear fitting rule.

// Target: probability-and-statistics.bigb

With fixed covariates and independent new responses having the same marginal distributions as the training responses, expanding squared losses cancels their second moments. Independence makes $\mathbb{E}(Y_i^{\mathrm{new}}{\hat{Y}}_i)=\mathbb{E}{Y}_i\mathbb{E}{\hat{Y}}_i$, yielding the displayed identity. For a fixed linear smoother $\hat{Y}=HY$ with covariance ${\sigma }^{2}I$, it reduces to $2{\sigma }^2\mathrm{tr}(H)/n$, linking prediction optimism to effective degrees of freedom.

// Target: analysis.bigb

Data leakage occurs when information unavailable at prediction time, including information from nominally held-out observations, influences model fitting or selection. It makes estimated out-of-sample performance optimistically biased.

Dataset shift means that the joint distribution of predictors and responses differs between training and deployment data.

A regression function is a conditional expected response. For binary labels it equals the conditional probability of class one.

Principal component analysis finds orthogonal directions of greatest sample variance. Its first loading vector is a unit eigenvector corresponding to the largest eigenvalue of the sample covariance matrix.

Regress a response on selected principal component scores of centered predictors. If $X=UD{V}^T$ is a singular value decomposition and the first $q$ positive-singular-value components are retained, the retained score matrix is $U_q{D}_q$ and the fitted coefficient in predictor coordinates is $V_q{D}_q^{-1}{U}_q^{T}y$. The fitted centered response is $U_q{U}_q^{T}y$: this follows by applying the least-squares normal equations to the orthogonal score columns. Truncating small singular directions limits variance amplification, but directions of large predictor variance need not be those best predicting the response; select $q$ using prediction assessment rather than explained predictor variance alone.

A scree plot graphs ordered covariance or correlation eigenvalues against component number. A transition from a steep decline to a flatter tail suggests a dimension, but this is a diagnostic rather than a proof of the number of latent factors. In factor analysis, it should be combined with model fit, substantive interpretation and uncertainty assessment.

Two conventions occur for principal-component loadings: the unit eigenvector ${\ell }_j$ defining a principal component, and the scaled vector $r_j=\sqrt{{\lambda }_j}{\ell }_j$. For principal component analysis on a correlation matrix, the entries of the latter are the correlations between each standardized variable and the component scaled to variance one. Indeed, $\mathrm{Cov}(Z,{\ell }_j^{T}Z)=C{\ell }_j={\lambda }_j{\ell }_j$ and $\mathrm{Var}({\ell }_j^{T}Z)={\lambda }_j$. Thus $\Vert r_j{\Vert }^2={\lambda }_j$, whereas $\Vert {\ell }_j\Vert =1$; reporting the convention avoids confusing component directions with correlations.

A principal component is the projection of centered data onto a unit eigenvector of its covariance matrix. Ordering the components by their eigenvalues orders their variances. Repeated eigenvalues allow any orthonormal basis of the corresponding eigenspace.

A score is an observation's coordinate along a principal component direction. Center an observation by the fitted sample mean, apply the fitted variable scaling when using principal component analysis on a correlation matrix, then take its inner product with the unit principal component loading vector. New observations must use the training center and scaling. The score variance equals the corresponding eigenvalue when the covariance normalization is consistent.

A population principal component explains the displayed fraction of total variance. The first $q$ components explain ${\sum }_{j\le q}{\lambda }_j/{\sum }_j{\lambda }_j$. Comparing covariance and correlation analyses requires different denominators: the original variance sum for the former and the number of nonconstant standardized variables for the latter.

For positive marginal variances, standardizing each variable to variance one converts covariance-based PCA on $Z$ into correlation-based PCA on $X$. This makes the analysis insensitive to the original measurement units but changes the optimization criterion. Standardization gives small-variance variables the same marginal weight as large-variance variables; it is a modelling choice, not an automatic improvement.

The population principal components are centered linear compounds along an orthonormal eigenbasis of the covariance matrix, ordered by decreasing eigenvalue. They are uncorrelated and have variances ${\lambda }_j$. Each successive direction maximizes variance subject to unit length and orthogonality to earlier directions. Uncorrelatedness does not imply independence unless additional distributional assumptions hold.

Let the columns of a centered design matrix $X\in {\mathbb{R}}^{n\times p}$ be variables and write $X^{T}X=V\Lambda V^T$ by the spectral theorem for real symmetric matrices. The columns $u_i$ of $U=XV$ are the sample principal-component score vectors. They satisfy $u_i^T{u}_j={\Lambda }_{ij}$, so distinct score vectors have zero sample covariance and $u_i$ has sample variance ${\Lambda }_{ii}/n$ under the divisor-$n$ convention.

When ${\Lambda }_{ii}>0$, the normalized sample principal component $w_i=u_i/\sqrt{{\Lambda }_{ii}}$ has Euclidean norm one. The vectors $w_i$ form an orthonormal set spanning the column space of a full-column-rank centered design matrix.

Classification predicts a discrete label from observed covariates.

The misclassification rate is the expected zero-one loss of a classification in statistical learning rule. Its empirical version is the fraction of incorrectly labeled observations. Evaluation on the fitting sample gives a training error; evaluation on an independent test set estimates prediction error. For a majority-class leaf of a classification tree, the training error count is $n_t-{\max }_k{n}_{tk}$, unlike classification-tree deviance, which also uses the fitted class probabilities.

A classification tree partitions predictor space recursively and attaches a class probability vector to each terminal region. For a categorical response, a fitted leaf commonly uses its observed class proportions and predicts their largest entry. Binary numerical splits compare one coordinate with a threshold. Growing a tree greedily by reducing classification-tree deviance gives an interpretable rule but does not globally optimize the tree. Cost-complexity tree pruning and cross-validation control complexity and assess prediction error.

Pruning balances a tree's empirical loss $R(T)$ against the number of leaves. Increasing $\alpha$ favors a nested sequence of smaller subtrees, obtained by removing splits with the smallest effective cost per removed leaf. Choose complexity using cross-validation or a validation set, and assess the selected tree on a separate test set. The loss can be deviance or misclassification loss; the chosen convention must be specified.

Maximizing a node's multinomial likelihood gives class probabilities ${\hat{p}}_{tk}=n_{tk}/n_t$. Twice the negative maximized log-likelihood relative to perfect classification is the displayed deviance, with $0\log 0=0$. It equals twice the node size times its Shannon entropy. A split is chosen by maximizing $D(t)-D(t_L)-D(t_R)$, and tree deviance is the sum over terminal nodes. The training misclassification count is instead ${\sum }_t(n_t-{\max }_k{n}_{tk})$; the two criteria are different.

Classification in statistical learning with two possible labels. Given the conditional probability $\eta (x)=\mathbb{P}(Y=1\mid X=x)$, a Bayes classifier under zero-one loss predicts one exactly when $\eta (x)\ge 1/2$, with arbitrary decisions at ties.

The false positive rate is $\mathbb{P}(\text{positive result}\mid \text{truly negative class})$. In screening, it measures erroneous positives among genuinely disease-free individuals.

A decision boundary is the subset of covariate space across which a classifier changes its predicted class.

For binary classification under zero-one loss, the Bayes decision boundary is the set on which the two conditional class probabilities are equal.

A majority-vote ensemble predicts the class selected by more constituent classifiers than any other class.

A confusion matrix cross-tabulates true classes against predicted classes. Its diagonal entries count correct predictions and its off-diagonal entries describe the kinds of classification errors.

A classification and regression tree recursively splits covariate space into axis-aligned regions and assigns a constant prediction to each terminal region.

Recursive partitioning repeatedly selects a split of one current region, then applies the same splitting rule independently to the resulting child regions.

A surrogate split uses another predictor to approximate the left-versus-right assignments made by a tree node's primary split. It can route observations whose primary predictor is missing. Surrogates are ranked by agreement with the primary split on observations where both predictors are known, accounting for trivial majority assignments. This differs from simply filling a missing predictor with its sample mean.

For class proportions $p_1,\dots ,p_K$ in a tree node, Gini impurity is $1-{\sum }_k{p}_k^2$. For two classes it is twice $p(1-p)$; omitting the constant factor does not change the selected split.

A random forest aggregates deeply grown decision trees fitted to bootstrap samples, while restricting each split to a random subset of predictor coordinates. Classification uses a majority vote over the trees.

The out-of-bag error predicts each training observation using only trees whose bootstrap samples omitted it, then averages the resulting losses.

The conditional class probability for class $k$ isA Bayes classifier selects a class maximizing this probability.

A sequence of classifiers $h_n$ is risk-consistent when its classification risk converges to the Bayes risk as the training sample size tends to infinity.

The K-nearest neighbors algorithm predicts from the labels or responses of the $K$ training observations closest to the query under a chosen distance.

With fixed neighbor positions, independent equal-variance errors and weights $w,(1-w)/2,(1-w)/2$, the displayed variance is minimized at $w=1/3$, with value ${\sigma }^2/3$. Its bias is $w{m}_1+(1-w)(m_2+m_3)/2-m(x)$. Larger nearest-point weight often reduces smoothing bias, but neither the bias nor the variance is universally monotone over $0<w<1$. If the nearest point is the target, the squared bias is $(1-w{)}^2[(m_2+m_3)/2-m(x){]}^2$.

// Target: probability-and-statistics.bigb

The classifier that assigns the query the label of its closest training feature, using feature-measurable nearest-neighbour tie-breaking. Its misclassification risk need not converge to Bayes risk, even with infinitely many independent and identically distributed random variables as training data.

For binary classification, write $\eta (x)=\mathbb{P}(Y=1\mid X=x)$ and assume feature-measurable nearest-neighbour tie-breaking. If $\mathbb{E}|\eta (X_{(1)}(X))-\eta (X)|\to 0$, then the expected conditional misclassification risk of the one-nearest-neighbour classifier tends to $\mathbb{E}[2\eta (X)(1-\eta (X))]$. Couple each label and an oracle label using one uniform variable. Their disagreement has the preceding expected absolute difference. The oracle and the test label are conditionally independent Bernoulli random variables of parameter $\eta (X)$, so their disagreement probability is exactly $2\eta (X)(1-\eta (X))$. The difference of classification in statistical learning error probabilities is at most the coupling disagreement.

Under the hypotheses for the one-nearest-neighbour asymptotic risk, the limiting risk satisfies $R^{\ast }\le R_{\infty }\le 2R^{\ast }$. Put $a=\min (\eta ,1-\eta )$: then $a\le 2a(1-a)\le 2a$ since $0\le a\le 1/2$. Integrate, using the definition of Bayes risk. Constant $\eta =1/4$ gives limiting risk $3/8$ and Bayes risk $1/4$, so the lower comparison is not generally equality.

For a fixed query, order training features by their distances and break equal distances by a rule depending only on features, such as original index. If auxiliary uniforms are independent of the features, the uniform selected by the resulting index remains uniform conditional on all features: sum over the possible selected indices. Label-dependent tie-breaking can invalidate Bernoulli coupling by a shared uniform random variable and the usual one-nearest-neighbour asymptotic risk.

The L-nearest-neighbour classifier estimates each class probability by its empirical frequency among the $L$ nearest training covariates and predicts a class of greatest estimated probability.

One-nearest-neighbour classification assigns a query the label of its single nearest training observation. Under local regularity, its limiting conditional error is $1-{\sum }_k{p}_k(x{)}^2$.

For binary classification with posterior estimate ${\hat{p}}_1$, the plug-in classifier satisfies

A support vector machine chooses a separating decision function by trading a wide geometric margin against hinge-loss violations.

A kernel support vector machine trains a support vector machine using a positive-definite kernel instead of explicit feature coordinates. The kernel trick evaluates all required feature inner products through the training Gram matrix; prediction is ${\sum }_i{\alpha }_i{y}_{i}k(x_i,x)+b$. Nonzero dual coefficients identify support vectors. An unpenalized intercept supplies the dual equality ${\sum }_i{\alpha }_i{y}_i=0$. The Reproducing kernel Hilbert space construction explains why a positive-semidefinite kernel defines valid feature geometry.

For signed training labels, a linear soft-margin support vector machine minimizeswith $C>0$. Its Karush-Kuhn-Tucker conditions give $w={\sum }_i{\alpha }_i{y}_i{x}_i$, ${\sum }_i{\alpha }_i{y}_i=0$, and $0\le {\alpha }_i\le C$.

For the kernel support vector machine objective $n^{-1}{\sum }_i(1-Y_i(\mu +K_i^T\alpha ){)}_{+}+\lambda {\alpha }^{T}K\alpha$, assume $Y_i\in \{-1,1\}$, $\lambda >0$, and that the kernel matrix $K$ is invertible. The subdifferential optimality equation givesThus strict margins greater than one force zero coefficients, while misclassified observations have nonzero coefficients. Invertibility matters: a singular kernel matrix allows coefficient changes in its null space without changing the fitted function or objective.

For a fixed $C$ in the unnormalized sum-of-slacks objective and unique training optimizers, deleting an observation with ${\alpha }_i=0$ preserves the Karush-Kuhn-Tucker conditions and the fitted decision function. It is correctly classified when held out. Consequently Leave-one-out cross-validation makes at most $s$ errors, where $s$ counts positive dual coefficients. With geometric support vectors, counting all points of signed margin at most one also gives the bound, without a nondegeneracy assumption.

A dual support vector has ${\alpha }_i>0$. Every such point has signed margin at most one, but a point exactly on the margin can have ${\alpha }_i=0$ in a degenerate solution. If ${\alpha }_i=0$, complementary slackness forces ${\xi }_i=0$, so the point is correctly classified with margin at least one.

For signed observations $(x_i,y_i)$, an affine hyperplane with score $f(x)=\alpha +x^T\beta$ is separating when $y_{i}f(x_i)>0$ for every observation.

The slack variables satisfy ${\xi }_i\ge \max \{0,1-y_{i}f(x_i)\}$ and quantify margin violations.

For a linear score $f(x)=\alpha +x^T\beta$, the decision boundary is $f(x)=0$.

For a linear support vector machine, the geometric margin lies between $f(x)=-1$ and $f(x)=1$ and has width $2/\Vert \beta \Vert$.

The two margin boundaries of a linear support vector machine are the parallel hyperplanes $f(x)=1$ and $f(x)=-1$.

A support vector is a training observation on or inside the fitted margin; it has a nonzero dual coefficient and can affect the fitted boundary.

The perceptron is an online linear classifier that adds a misclassified observation's signed feature vector to its current parameter vector.

Cross-validation estimates predictive performance by repeatedly fitting on one part of the data and evaluating on held-out observations. In K-fold cross-validation, divide training data into disjoint folds $I_1,\dots ,I_K$; $k(i)$ identifies observation $i$'s fold, and ${\hat{f}}^{(-k)}$ is fitted without fold $I_k$. The displayed average scores each observation using a rule that did not train on it. Weight by observation count when fold sizes differ. Leave-one-out cross-validation uses one observation per fold. ^[ref]

For the training error example, three folds contain indices $i$ with equal remainder modulo 3. Each fit uses eight observations. The squared-error loss averages for degrees 1, 2, and 7 are $0.724132$, $0.027192$, and $1.532790$ respectively. Select degree 2 and refit on all twelve training data observations. The folds reuse scarce data while challenging each candidate to predict unseen responses; they do not touch the final test set.

Any learned preprocessing, including variable selection, must also be fitted within each training fold to avoid data leakage. ^[ref] Repeated tuning can itself overfit the cross-validation score; nested cross-validation uses inner folds for model selection and outer folds for assessment of that entire procedure. ^[ref]

Folds use fewer training observations than the final refit, and their scores are dependent because training observations overlap. ^[ref] Splitting must match deployment: for time series forecasting, rolling-origin cross-validation trains on the past and evaluates later observations, preventing future information from entering a forecast. Ordinary random folds need not answer that forecasting question. ^[ref]

Nested cross-validation separates selection from assessment using two levels of held-out data. Within each outer training fold, inner cross-validation chooses a model or tuning parameters; the complete selected fitting procedure is then applied to that outer training fold and scored on its untouched outer evaluation fold. This estimates the performance of the model selection procedure, including its choices, rather than reusing its best inner score as an assessment. A final model can then be selected and fitted using all available training data. ^[ref]

Subtract the diagonal variance contribution from the derivative estimate's squared norm, then insert the result into asymptotic mean integrated squared error. The expected curvature estimate equals $R(f^{\prime \prime })+O(h^2)$, apart from a negligible finite-n term. This makes the resulting criterion asymptotically correct even though it is not an exactly unbiased estimate of integrated risk. Under standard conditions its selector is ratio-consistent with the optimal bandwidth and has relative fluctuation order $n^{-1/10}$.

Minimize the displayed criterion using a Leave-one-out cross-validation estimate in the second term. Its expectation is the exact mean integrated squared error minus $R(f)$, which does not depend on h. With standard smoothness and appropriate search ranges, the chosen bandwidth is asymptotically optimal. Its relative fluctuations are ordinarily of order $n^{-1/10}$, considerably larger than the estimation error achievable by well-chosen plug-in pilots.

For a fixed linear smoother, generalized cross-validation replaces the individual leverage corrections of Leave-one-out cross-validation by their average. It selects smoothing parameters by minimizing the displayed criterion. It approximates a predictive error criterion; it is not a universal exact estimate of future error and requires care when observation errors are dependent.

Leave-one-out cross-validation performs one fit with each observation held out in turn and averages the held-out losses.

If a linear smoother has fitted vector $\hat{Y}=HY$, its residual after fitting without observation $i$ isThe identity follows from a block matrix inverse or a rank-one inverse update.

K-fold cross-validation partitions observations into $K$ folds, trains $K$ times while holding out one fold at a time, and combines the held-out losses.

A neural network composes affine maps with nonlinear activation functions and learns their weights from data.

A binary threshold unit outputs one when its weighted input reaches its threshold and zero otherwise. Its hard threshold is not differentiable, so ordinary backpropagation cannot directly train the exact threshold model. Smooth activation functions can approximate its computation.

An activation function converts a unit's weighted input to its output. A sigmoid function, hyperbolic tangent or rectifier can give nonlinear features. Replacing all hidden activation functions by the identity collapses consecutive affine layers into a single affine map, removing the representational benefit of those hidden layers.