Statistical learning

Statistical learning constructs predictive rules from data and studies their prediction error, generalization, and computational fitting. In supervised learning, training data contain inputs $X$ and a response variable $Y$: regression analysis predicts a numerical response, while classification in statistical learning predicts a label. Unsupervised learning instead seeks structure without supplied responses, as in cluster analysis and principal component analysis. These tasks connect learning to probability and mathematical optimization. ^[ref]

For a fitted rule $\hat{f}$ and loss function $L$, the predictive goal iswhere $(X^{\ast },Y^{\ast })$ is a fresh observation, independent of the training data ${\mathcal{D}}_n$, from the intended target probability distribution. This prediction error fixes the fitted rule and averages over future observations; averaging again over training data gives the learning procedure's average performance. Its practical beauty is that a rule compresses past observations into useful future predictions. Its central difficulty is that excellent training error need not imply good generalization: cross-validation supports model selection, and a reserved test set assesses the resulting choice.

Representing data using fewer coordinates. A method must specify which properties it tries to preserve, such as reconstruction, local distances, prediction or temporal structure. Preserving most total variance need not preserve the information required by a task.

Estimating a solution of a deterministic equation through repeated noisy samples of its residual or update direction. Temporal-difference policy evaluation is an example. Convergence depends on stability, sampling, noise and step sizes, rather than merely on the existence of an unbiased sample.

A common condition along each repeatedly sampled component is ${\sum }_n{\alpha }_n=\infty$ and ${\sum }_n{\alpha }_n^2<\infty$. The first permits continuing correction; the second limits accumulated noise under the relevant stochastic-approximation hypotheses. The example ${\alpha }_n=1/n$ has a divergent harmonic sum and a convergent sum of squares. These conditions are not sufficient without suitable assumptions about the update dynamics and sampling.

Learning from targets or constraints derived from the data or their transformations rather than externally supplied semantic labels. The supervision source and permitted transformations are part of the hypothesis. Contrastive representation learning is one family of such methods.

Learning a representation by relating selected matching examples or views and contrasting them with other examples. The augmentation, matching and sampling rules determine which distinctions are retained. Brain-like prediction in a trained model does not establish the exact objective or plasticity mechanism in cortex.^[ref]

An estimate of the agreement achievable with a measured dataset given its reliability, variation across observations or participants, and the chosen comparison metric. Reaching it does not mean that a model captures every biological property; a more informative dataset can expose differences that the original data could not resolve.

The need to adapt to new evidence while preserving useful information from prior experience. Freezing everything prevents learning; changing shared representations without appropriate constraints can damage retention. Complementary learning systems and replay address aspects of this trade-off.

A severe loss of previously learned performance when updating a shared representation on new material. Overlapping parameters allow transfer but also let a new objective change computations needed for older examples. Interleaving and representation design can reduce the problem; the mere presence of a neural network does not imply inevitable catastrophic forgetting.

Constraints or preferences that guide generalization beyond observed training examples. Local convolution, weight sharing, a compositional grammar and a prior over programs supply different inductive biases. Their usefulness depends on the structure of the target problem.

Learning to choose actions from their consequences in an environment, with the objective of increasing expected reward or return. Reward does not normally specify a correct action label. State representation, exploration and temporal credit assignment are central computational requirements.

Learning about a distribution of possible future returns rather than only its mean. Multiple prediction channels can vary in optimism and sensitivity to positive and negative errors. Such population heterogeneity can refine a reward prediction error account without implying that every dopamine signal performs the same computation.

Using stored or regenerated experience for further learning, often interleaved with new observations. It can improve data reuse and retention, but its sampling distribution and fidelity affect what objective is learned. A software replay buffer is not a complete biological model of hippocampal reactivation.

Learning or using a model of transitions and outcomes to evaluate possible actions. It can support flexible replanning when the value of an outcome changes, but depends on the learned model and computational resources. Behavioural experiments must distinguish it from alternative strategies with similar observable predictions.

Learning cached values or action preferences without explicitly planning through a model of state transitions and outcomes. A basic actor-critic is an example. The absence of a transition model can limit rapid revaluation of an unvisited action after its outcome changes.

A value prediction is updated using a reward and a subsequent prediction: ${\delta }_t=r_{t+1}+\gamma V(s_{t+1})-V(s_t)$. The linear semi-gradient rule is $\Delta w=\alpha {\delta }_{t}x(s_t)$ for $V_w(s)=w^{\mathsf{T}}x(s)$; the target is held fixed for this update. Terminal-state value is zero in an episodic task.^[ref]

A sampled transition updates only the visited value: $V(S_t)\leftarrow V(S_t)+{\alpha }_t{\delta }_t$, where ${\delta }_t=R_{t+1}+\gamma V(S_{t+1})-V(S_t)$. Under the fixed policy, its conditional mean is $(T^{\pi }V)(s)-V(s)$. Standard tabular convergence requires adequate state visitation and step-size/noise conditions; this mean identity alone does not prove convergence for arbitrary function approximation.

A discrepancy between obtained or newly predicted reward and the reward previously expected. In a temporal-difference model it includes the next-state prediction, rather than being simply the immediate reward. A positive value signals a better-than-predicted outcome; a negative value signals a worse-than-predicted one.

An actor learns an action-selection policy while a critic learns a value prediction used to evaluate outcomes. A temporal-difference error can update both. With an accurate on-policy state-value function, its conditional mean given a state and action is that action’s advantage, supplying a useful policy-learning direction without a labelled correct action.

Analysis using patterns across multiple measured features rather than a single averaged response. In neuroimaging, classification or related methods can distinguish remembered content from spatial activity patterns. Independent testing and control of nuisance structure are essential to interpreting the result.

Training a statistical predictor on one modality and evaluating it on another to test information that generalises across input or output forms. Stimulus identity, response mapping, noise and held-out evaluation affect what a successful or failed result means.

Comparison of the pairwise relationships among measured or model response patterns. The same stimuli are presented to the systems being compared, and their representational dissimilarity matrices are related. A match can constrain representation without requiring one-to-one unit correspondence or establishing the underlying learning rule.

A matrix whose entry $D_{ij}$ describes how differently a system responds to stimuli $i$ and $j$. One convention is $D_{ij}=1-\mathrm{corr}(z_i,z_j)$ when the response vectors have nonzero variance. Such a dissimilarity need not satisfy every metric axiom. Model comparison must control fitted transformations and use held-out data.

Learning through a sequence of decisions and observations, assessing performance by comparison with a benchmark decision or rule. A multi-armed bandit is a model in which only the reward of the selected action is observed.

A decision principle that maximizes possible reward jointly over an action and a parameter in a confidence set. On the event that the set contains the true parameter, the reward predicted for this selected pair is at least the true optimal reward. This inequality bounds pseudoregret by the width of the confidence set in the selected action's direction. For an $L$-Lipschitz continuous reward link and a radius-$\beta$ confidence ellipsoid, Cauchy-Schwarz inequality and the triangle inequality bound that width by $2L\beta \sqrt{x^T{V}^{-1}x}$.

A sequential decision model in which each round selects an action and observes its reward, balancing information gathering against immediate reward. For fixed action means $m(x)$, the pseudoregret compares selected actions with an action maximizing $m$. In models with a link function and parameter $\theta$, rewards may have means $m_{\theta }(x)=\mu (⟨x,\theta ⟩)$.

Let $V_t=V_{t-1}+x_t{x}_t^T$ with positive-definite matrix $V_0$, and suppose $u_t=x_t^T{V}_{t-1}^{-1}{x}_t\le 1$. The matrix determinant lemma gives $\det V_t/\det V_{t-1}=1+u_t$. Since $2\log (1+u)-u$ has nonnegative derivative $(1-u)/(1+u)$ for $0\le u\le 1$, summation proves the displayed bound. In particular, $V_0=\lambda I$, $\lambda \ge 1$, and $\Vert x_t{\Vert }_2\le 1$ ensure $u_t\le 1$. Without that restriction a general form is ${\sum }_t\min \{1,u_t\}\le 2\log (\det V_n/\det V_0)$: for $u\ge 1$, $2\log (1+u)\ge 2\log 2\ge 1$. The bound connects widths of confidence ellipsoids to growth of a determinant.

The cumulative difference between the mean reward of an optimal fixed action and the mean rewards of the selected actions. For adaptive random actions, this sum is itself a random variable; taking its expected value gives expected pseudoregret. Observation noise is absent from the sum, although earlier noise can influence later actions.

A predictive dependence in which the effect of one input feature changes with another. A product term is a simple example that cannot be represented by a purely additive linear predictor.

Prediction of several labels that can occur together. Independent sigmoid outputs do not impose the sum-to-one constraint used for mutually exclusive classes.

Classification with more than two mutually exclusive outcome classes. A softmax output can represent a normalized distribution over the classes.

Representation of one categorical outcome by a vector with one component equal to one and all other components zero.

A learned method for predicting certain cellular perturbation responses from single-cell RNA sequencing data. Generalization to evaluated held-out cell types or conditions does not establish predictive validity for all interventions.^[ref]

Supervised learning fits a prediction rule from training data containing both inputs and associated target responses. Regression analysis predicts numerical responses; classification in statistical learning predicts class labels. A loss function specifies which prediction errors matter, and generalization concerns performance beyond the observations used to fit the rule.

Learning driven by a discrepancy between a target and the network’s actual output. The architecture must supply both a meaningful teaching signal and a mechanism for assigning that signal to the connections responsible for the error. This is distinct from error correction during retrieval in a fixed associative memory.

For a linear output $y=Wx$ and squared loss $L=\Vert t-y{\Vert }^2/2$, gradient descent gives $\Delta W=\eta (t-y)x^{\mathsf{T}}$. Indeed $\partial L/\partial W_{ij}=-(t_i-y_i)x_j$. On the same example, $e^{\prime }=(1-\eta \Vert x{\Vert }^2)e$, proving strict error reduction when $e\ne 0$ and $0<\eta \Vert x{\Vert }^2<2$.

Generalization is the ability of a fitted prediction rule to perform well on new observations under a specified target probability distribution. It is assessed through prediction error rather than training error. The target distribution and use of data are part of the question: dataset shift can change performance, and data leakage can make an apparent assessment misleading.

Generalizing known components or operations to combinations not encountered during training. A held-out combination split tests a different ability from random train–test splitting when both sets contain nearly identical constructions. A failure of a particular neural training setup does not prove a limitation of every network architecture.^[ref]

Learning structure from data without supplied target labels. Examples include extracting principal components, clustering and learning representations that efficiently reconstruct inputs.

Learning transformation-tolerant representations by combining input-feature overlap, current-activity Hebbian learning and synaptic competition. A new view partly activates an output already learned for an overlapping view, then strengthens its new features onto that output. Chains of overlapping views can link more distant transformations without an explicit temporal trace; dense enough overlap and suitable competition remain conditions.^[ref]

Units compete to respond to an input, and the winning unit or sparse winning set strengthens its representation of that input. A prototype update is $w_k\leftarrow w_k+\eta (x-w_k)$ for the winner $k$, with $0<\eta <1$. The winner moves toward the sample: $w_k^{\prime }-x=(1-\eta )(w_k-x)$. Competition and constraints prevent all units from learning the same unrestricted average.

Independently varying companions can make an individual object’s features more consistently related than the features of any particular multi-object conjunction. Competitive networks can exploit this difference to learn separate object representations without being shown each object in isolation. The result depends on the training distribution; a fixed co-occurring group can instead be learned as a conjunction.^[ref][ref]

Construct low-dimensional coordinates representing a dissimilarity matrix. Different methods preserve different features: classical multidimensional scaling approximates a centered Gram matrix, while other methods minimize discrepancies between fitted and supplied distances or preserve distance ordering. The method and dimension determine what geometric claims the resulting display supports.

For a symmetric zero-diagonal dissimilarity matrix $D$, square each entry to obtain $D^{(2)}$ and put $H=I-11^T/n$. Take the eigenvectors of $G=-H{D}^{(2)}H/2$ corresponding to its largest positive eigenvalues. Multiplying them by square roots of these eigenvalues gives coordinates. If $D$ contains Euclidean distances, $G$ is a centered Gram matrix, so retaining all positive eigenvalues reconstructs those distances exactly. Retaining fewer approximates the Gram matrix, not generally the sum of squared errors in raw distances. Negative eigenvalues signal failure of an exact Euclidean representation. Coordinates are unchanged in meaning by orthogonal transformations.

For symmetric nonnegative $D$ with zero diagonal, a centered Euclidean realization implies $D_{ij}^2=G_{ii}+G_{jj}-2G_{ij}$ and hence the displayed double-centering identity. Conversely, if the double-centered matrix is positive semidefinite, factor it as $X{X}^T$. Its reconstructed squared distances equal the original squared distances: their difference has zero double-centering, so is of the form $1a^T+a{1}^T$; the zero diagonal forces $a=0$. The minimum exact embedding dimension is $\mathrm{rank}G$.

Represent inputs accurately using few strongly active coefficients, for example by minimizing $\Vert x-Az{\Vert }^2/2+\lambda {\sum }_i|z_i|$. Learning basis functions on natural images can yield localized oriented filters resembling simple-cell receptive fields. This demonstrates one possible functional account of selectivity, not a proof that cortex implements that exact objective or learning algorithm.

Activity-dependent strengthening associated with coactivation of an input and its postsynaptic output. The idealized update $\Delta w_i\propto y{x}_i$ requires normalization, competition or other constraints to prevent indefinite weight growth.

In Hebbian memory weights for a Hopfield network, the local update $\Delta w_{ij}=\eta {\xi }_i{\xi }_j$ records a pairwise association, where $\eta >0$ and ${\xi }_i,{\xi }_j\in \{-1,1\}$. Equal signs strengthen a positive coupling; opposite signs favor a negative coupling. Summing such updates gives a symmetric matrix of pairwise associations. This is a storage prescription: it does not by itself compare the generated probability distribution with the training data, as Boltzmann machine learning does. Bipolar activity also makes joint inactivity contribute positively; this encoding is a mathematical idealization, not a literal statement about inactive biological cells.

A local association rule uses a decaying activity trace to connect representations encountered at nearby times. One implementation has ${\bar{y}}_i(t)=\rho {\bar{y}}_i(t-1)+(1-\rho )y_i(t)$ and $\Delta w_{ij}=\eta {\bar{y}}_i(t-1)x_j(t)$, followed by synaptic normalization. Temporal continuity can link distinct views of one object, but a trace spanning an identity change can create a false association.^[ref]

With $y=w\cdot x$, the squared weight norm $s=|w{|}^2$ satisfies $\tau \dot{s}=2y^2(1-\alpha s^2)$. It moves monotonically toward $s_{\ast }={\alpha }^{-1/2}$ without crossing it. Explicitly $(s-s_{\ast })/(s+s_{\ast })$ equals its initial value times $\exp [-(4\sqrt{\alpha }/\tau ){\int }_0^{t}y(u{)}^{2}du]$. Persistent excitation therefore drives $|w|$ to ${\alpha }^{-1/4}$; without excitation it can remain stationary. This nonlinear normalization avoids the unbounded amplification of unmodified Hebbian learning.

For a linear output $y=w^{T}x$, the second term limits the growth produced by the Hebbian term. On the averaged dynamics with $C=\mathbb{E}[x{x}^T]$, $\tau \dot{w}=Cw-\alpha (w^{T}Cw)w$. For $\alpha >0$, positive-eigenvalue equilibria have norm $1/\sqrt{\alpha }$ and align with eigenvectors of $C$. If inputs are centered, $C$ is their covariance matrix.

At the equilibrium $v_i/\sqrt{\alpha }$, linearization along orthonormal eigenvectors of $C$ gives the displayed rates, with the first equation applying to $j\ne i$. A simple positive largest eigenvalue makes its two normalized eigenvectors locally asymptotically stable; smaller eigenvalues have an unstable direction. Repeated largest eigenvalues give a manifold of equilibria with neutral directions. Generic convergence also needs a nonzero initial component in the largest eigenspace. The squared norm satisfies $\tau d\Vert w{\Vert }^2/dt=2(w^{T}Cw)(1-\alpha \Vert w{\Vert }^2)$.

Cluster analysis groups observations using similarities or dissimilarities without supplied class labels. A distance choice specifies which features and scales matter. Agglomerative hierarchical clustering merges groups recursively, producing partitions at different resolutions rather than one intrinsically determined number of groups.

A cluster in cluster analysis is a nonempty group of observations treated as similar under the chosen representation and dissimilarity. A partition assigns each observation to exactly one cluster; agglomerative hierarchical clustering supplies a nested sequence of such partitions. An algorithmic cluster need not correspond to a distinct probabilistic population.

For a prescribed number of groups, minimize within-cluster squared Euclidean distances. At fixed assignments, differentiating the objective gives the mean as the optimal centroid. At fixed centroids, assigning each point to a nearest centroid minimizes its contribution. Alternating these steps cannot increase the objective but can converge to a local optimum, so multiple starts are useful. Empty clusters and ties need an explicit handling rule. The method favors roughly spherical groups and depends on variable scaling.

For $n$ distinct observations on the real line and $1\le K\le n$, nearest-centre assignment produces contiguous blocks of the sorted observations because each Voronoi cell is an interval. The arithmetic means of distinct ordered blocks are strictly ordered, preserving this property at subsequent assignments. Initialize at $K$ distinct observed values so that the initial assignment has no empty cluster.

The empty-cluster repair in finite termination of K-means clustering also preserves contiguous blocks: an observation farthest from a centre over its assigned block lies at one of that block's endpoints, so transferring it to a new singleton splits the block into two contiguous blocks.

There are $\left(\genfrac{}{}{0ex}{}{n-1}{K-1}\right)$ possible partitions into $K$ nonempty blocks: choose $K-1$ cuts among the $n-1$ gaps. Under the stable tie rule in finite termination of K-means clustering, every round after the first strictly reduces the objective until stopping, so a block partition cannot recur. The number of assignment, repair, and centre-update rounds is therefore at most $\left(\genfrac{}{}{0ex}{}{n-1}{K-1}\right)\le n^{K-1}$. Check stability after each update without an additional update round; for $K=1$, one update suffices.

For a finite dataset, alternate nearest-centre assignment and replacement of each centre by the centroid of its assigned observations. Keep the current assignment whenever an observation is tied between nearest centres. A changed assignment then strictly decreases the within-cluster sum of squared Euclidean distances; updating centres cannot increase it by the centroid sum-of-squares identity.

After each centre update, the centres are determined by the current partition of a set of observation indices. There are only finitely many such partitions, so a strictly improving sequence cannot revisit one and must stop. A convenient exact stopping test is to check after each update whether every observation is still assigned to a nearest centre. This proof is for algorithm termination; it does not guarantee a global minimum.

Empty clusters need an explicit convention. To retain exactly $K\le n$ nonempty clusters, transfer an observation from a cluster with at least two members to each empty cluster, making it a singleton centred at that observation. This cannot increase the objective: the transferred observation now has squared distance zero, and no other contribution changes before centre updating. A deterministic choice is an eligible observation farthest from its currently assigned centre, with ties resolved by index. After the first round, an empty cluster can arise only during a strictly improving reassignment, so these repairs preserve the finite-termination argument.

For two distinct centroids in a Euclidean space, equal squared Euclidean distances expand to the displayed affine hyperplane. It is their perpendicular bisector; each closed half-space consists of points at least as close to the centroid on that side. This proves weak linear separability after a two-centroid nearest-assignment step, allowing ties on the boundary. Coincident centroids are an essential exception: every point ties and the distance comparison supplies no nonzero normal vector.

For a nonempty finite collection of Euclidean vectors, its sample mean $\bar{x}=n^{-1}{\sum }_i{x}_i$ uniquely minimizes the sum of squared Euclidean distances to a proposed centroid $z$. Expand each squared Euclidean norm around $\bar{x}$ and use ${\sum }_i(x_i-\bar{x})=0$ to cancel the cross term. The displayed identity leaves a nonnegative term $n\Vert z-\bar{x}{\Vert }^2$, equal to zero exactly at $z=\bar{x}$.

The index measures similarity of finite sets by shared presence relative to total presence. For binary profiles it ignores joint absences, unlike the simple matching coefficient. When both sets are empty, define the similarity to be one. The complementary Jaccard distance measures dissimilarity.

The dissimilarity is the number of mismatched presences divided by the number of coordinates present in at least one profile. It differs from normalized Hamming distance because coordinates absent from both profiles do not enter its denominator. Define the distance of two empty sets to be zero.

A dissimilarity matrix records pairwise separation of observations or profiles. Usually it is symmetric, nonnegative and zero on the diagonal. A metric space additionally requires the triangle inequality and separation of distinct profiles. Agglomerative hierarchical clustering can be applied to many dissimilarity matrices that are not Euclidean distance matrices; the claimed geometry must be checked separately.

For binary vectors, this similarity counts agreements on both presences and absences. Its complement is the normalized Hamming distance. Whether joint absences should count as similarity depends on the application; rare-presence data may call for a different coefficient.

The displayed distance is the normalized Hamming distance. Each coordinate mismatch between $x,z$ implies a mismatch between $x,y$ or $y,z$, so summing the indicator inequalities proves the triangle inequality. Nonnegativity, symmetry and separation follow immediately. It is a metric on profiles; on separately labelled individuals with identical profiles it is a pseudometric.

Start with singleton clusters and repeatedly merge the two clusters with the smallest chosen intercluster dissimilarity. Single-linkage clustering uses the smallest cross-pair distance; complete-linkage clustering uses the largest. Merge heights and sensitivity to ties help assess a proposed number of groups.

Ward clustering merges the pair causing the smallest increase in within-cluster sum of squares. Expanding each cluster's squared deviations about the merged mean gives the displayed increase. Its Euclidean sum-of-squares interpretation must not be asserted for arbitrary non-Euclidean dissimilarities.

Average linkage merges the pair with the smallest mean cross-pair dissimilarity. This differs from the minimum used by single-linkage clustering and the maximum used by complete-linkage clustering. The convention here weights individual cross-pairs equally; weighting already formed clusters equally is a different algorithm.

The min and max cross-pair definitions of single-linkage clustering and complete-linkage clustering remain meaningful for a symmetric nonnegative dissimilarity matrix that violates the triangle inequality. Such a calculation gives a valid algorithmic dendrogram, but does not turn the original matrix into Euclidean distances. A triangle-inequality counterexample identifies that source defect without requiring invented replacement data.

A dendrogram displays successive cluster merges, with the height of each merge equal to its intercluster dissimilarity. Cutting at a height gives a partition. Single-linkage clustering measures the closest cross-pair and can form chains, while complete-linkage clustering measures the farthest cross-pair and favors smaller within-cluster diameters.

The farthest cross-pair distance determines the next merge. It favors groups with small diameter and reduces the chaining effect of single-linkage clustering, but can be sensitive to isolated extreme observations. Ties can affect intermediate partitions.

The nearest cross-pair distance determines the next merge. At a threshold, clusters are connected components of the graph joining pairs below that threshold. This handles nonconvex shapes but can join otherwise separate groups through a chain of intermediate observations.

A validation set is held out from parameter fitting and used to choose models or tuning parameters. A separate test set is then needed for an unbiased final evaluation after selection.

A test set is held out from both parameter fitting and model selection and is used to estimate the final selected procedure's prediction error. A validation set influences the choice; a test set assesses that choice after it is fixed. The distinction concerns how observations are used, not their file names.

For a fixed rule and an independent test set of size $m$ from the target probability distribution,This test error is an estimate, with sampling uncertainty; it is not a guarantee for each future prediction. Dataset shift changes what performance the estimate represents.

In the overfitting example, all degree choices are fixed in advance, cross-validation uses only training data, and the selected degree is locked before evaluating 400 separate points. Displaying the other curves afterward explains their behavior; it does not authorize selecting by those scores. Repeatedly inspecting a test set and changing the model makes it part of model selection, introducing data leakage and potentially optimistic assessment. ^[ref]

Prediction error is the expected loss of a fitted rule on a new observation from the target distribution.

Test error is the average loss on an independent test set and estimates prediction error when the test set has not influenced model fitting or selection.

For a predictor $\hat{f}$ and an independent response $Y^{\ast }$ at covariate $X^{\ast }$, mean squared prediction error is $\mathbb{E}[(Y^{\ast }-\hat{f}(X^{\ast }){)}^2]$.

Training error is the average loss function evaluated on the observations used to fit the prediction rule. It is the empirical risk of the fitted rule on its own training data. With squared-error loss, it is the average squared regression residual, measuring fit to observed responses rather than performance on fresh responses. Flexible fitting usually makes it optimistically biased for prediction error, as quantified by prediction optimism.

For a reproducible polynomial regression example, take $x_i=-1+2i/11$, $i=0,\dots ,11$, and the known noiseless target $f(x)=1+x/2+2x^2$. Construct responses $y_i=f(x_i)+e_i$ with fixed disturbancesOrdinary least squares over polynomials of polynomial degree at most 1, 2, and 7 gives training errors $0.518937$, $0.015784$, and $0.009257$. The monotone improvement follows because these candidate families are nested: every smaller-degree fit remains available in the larger family. This argument concerns minimum training error, not generalization, and assumes the fitting mathematical optimization is solved. Degrees 2 and 7 fitted instead to the exact noiseless responses $f(x_i)$ have mathematically zero training error; the script obtains approximately $5.31\times 10^{-31}$ and $7.82\times 10^{-31}$ from rounding. Extra flexibility alone does not force overfitting.

Prediction optimism is the difference between expected squared prediction error on new independent responses at the training covariates and expected training error. Reusing responses to fit and assess a model tends to make training error too favorable. The covariance formula for prediction optimism measures this reuse without assuming a linear fitting rule.

// Target: probability-and-statistics.bigb

With fixed covariates and independent new responses having the same marginal distributions as the training responses, expanding squared losses cancels their second moments. Independence makes $\mathbb{E}(Y_i^{\mathrm{new}}{\hat{Y}}_i)=\mathbb{E}{Y}_i\mathbb{E}{\hat{Y}}_i$, yielding the displayed identity. For a fixed linear smoother $\hat{Y}=HY$ with covariance ${\sigma }^{2}I$, it reduces to $2{\sigma }^2\mathrm{tr}(H)/n$, linking prediction optimism to effective degrees of freedom.

// Target: analysis.bigb

Data leakage occurs when information unavailable at prediction time, including information from nominally held-out observations, influences model fitting or selection. It makes estimated out-of-sample performance optimistically biased.

Dataset shift means that the joint distribution of predictors and responses differs between training and deployment data.

A regression function is a conditional expected response. For binary labels it equals the conditional probability of class one.

Principal component analysis finds orthogonal directions of greatest sample variance. Its first loading vector is a unit eigenvector corresponding to the largest eigenvalue of the sample covariance matrix.

Let $W,S$ be independent balanced variables with values $-1,1$, and let $X=(aW,bS{)}^T$ with $a>b>0$ and class $Y=S$. Then $E[X]=0$ and $\mathrm{Cov}(X)=\mathrm{diag}(a^2,b^2)$. The first principal component is the first coordinate and explains $a^2/(a^2+b^2)$ of total variance. Yet it is independent of $Y$, so even the best classifier using only that coordinate has accuracy $1/2$. The second coordinate determines $Y$ exactly. Taking $a/b$ arbitrarily large makes the retained variance approach one without improving class accuracy. Adding a constant baseline to each coordinate preserves the centered result and permits positive neural rates.

Regress a response on selected principal component scores of centered predictors. If $X=UD{V}^T$ is a singular value decomposition and the first $q$ positive-singular-value components are retained, the retained score matrix is $U_q{D}_q$ and the fitted coefficient in predictor coordinates is $V_q{D}_q^{-1}{U}_q^{T}y$. The fitted centered response is $U_q{U}_q^{T}y$: this follows by applying the least-squares normal equations to the orthogonal score columns. Truncating small singular directions limits variance amplification, but directions of large predictor variance need not be those best predicting the response; select $q$ using prediction assessment rather than explained predictor variance alone.

A scree plot graphs ordered covariance or correlation eigenvalues against component number. A transition from a steep decline to a flatter tail suggests a dimension, but this is a diagnostic rather than a proof of the number of latent factors. In factor analysis, it should be combined with model fit, substantive interpretation and uncertainty assessment.

Two conventions occur for principal-component loadings: the unit eigenvector ${\ell }_j$ defining a principal component, and the scaled vector $r_j=\sqrt{{\lambda }_j}{\ell }_j$. For principal component analysis on a correlation matrix, the entries of the latter are the correlations between each standardized variable and the component scaled to variance one. Indeed, $\mathrm{Cov}(Z,{\ell }_j^{T}Z)=C{\ell }_j={\lambda }_j{\ell }_j$ and $\mathrm{Var}({\ell }_j^{T}Z)={\lambda }_j$. Thus $\Vert r_j{\Vert }^2={\lambda }_j$, whereas $\Vert {\ell }_j\Vert =1$; reporting the convention avoids confusing component directions with correlations.

In principal component analysis, the matrix whose columns are the chosen orthonormal eigenvectors of a covariance matrix, ordered by decreasing eigenvalues. Under this convention the principal component scores of centered data are $Z=X_{c}V$. Some literature instead scales the vectors by square roots of eigenvalues; the convention should be stated explicitly.

A principal component is the projection of centered data onto a unit eigenvector of its covariance matrix. Ordering the components by their eigenvalues orders their variances. Repeated eigenvalues allow any orthonormal basis of the corresponding eigenspace.

A score is an observation's coordinate along a principal component direction. Center an observation by the fitted sample mean, apply the fitted variable scaling when using principal component analysis on a correlation matrix, then take its inner product with the unit principal component loading vector. New observations must use the training center and scaling. The score variance equals the corresponding eigenvalue when the covariance normalization is consistent.

A population principal component explains the displayed fraction of total variance. The first $q$ components explain ${\sum }_{j\le q}{\lambda }_j/{\sum }_j{\lambda }_j$. Comparing covariance and correlation analyses requires different denominators: the original variance sum for the former and the number of nonconstant standardized variables for the latter.

For positive marginal variances, standardizing each variable to variance one converts covariance-based PCA on $Z$ into correlation-based PCA on $X$. This makes the analysis insensitive to the original measurement units but changes the optimization criterion. Standardization gives small-variance variables the same marginal weight as large-variance variables; it is a modelling choice, not an automatic improvement.

The population principal components are centered linear compounds along an orthonormal eigenbasis of the covariance matrix, ordered by decreasing eigenvalue. They are uncorrelated and have variances ${\lambda }_j$. Each successive direction maximizes variance subject to unit length and orthogonality to earlier directions. Uncorrelatedness does not imply independence unless additional distributional assumptions hold.

Let the columns of a centered design matrix $X\in {\mathbb{R}}^{n\times p}$ be variables and write $X^{T}X=V\Lambda V^T$ by the spectral theorem for real symmetric matrices. The columns $u_i$ of $U=XV$ are the sample principal-component score vectors. They satisfy $u_i^T{u}_j={\Lambda }_{ij}$, so distinct score vectors have zero sample covariance and $u_i$ has sample variance ${\Lambda }_{ii}/n$ under the divisor-$n$ convention.

When ${\Lambda }_{ii}>0$, the normalized sample principal component $w_i=u_i/\sqrt{{\Lambda }_{ii}}$ has Euclidean norm one. The vectors $w_i$ form an orthonormal set spanning the column space of a full-column-rank centered design matrix.

Classification predicts a discrete label from observed covariates.

The fraction of predicted class labels that agree with the observed labels.

The misclassification rate is the expected zero-one loss of a classification in statistical learning rule. Its empirical version is the fraction of incorrectly labeled observations. Evaluation on the fitting sample gives a training error; evaluation on an independent test set estimates prediction error. For a majority-class leaf of a classification tree, the training error count is $n_t-{\max }_k{n}_{tk}$, unlike classification-tree deviance, which also uses the fitted class probabilities.

A classification tree partitions predictor space recursively and attaches a class probability vector to each terminal region. For a categorical response, a fitted leaf commonly uses its observed class proportions and predicts their largest entry. Binary numerical splits compare one coordinate with a threshold. Growing a tree greedily by reducing classification-tree deviance gives an interpretable rule but does not globally optimize the tree. Cost-complexity tree pruning and cross-validation control complexity and assess prediction error.

Pruning balances a tree's empirical loss $R(T)$ against the number of leaves. Increasing $\alpha$ favors a nested sequence of smaller subtrees, obtained by removing splits with the smallest effective cost per removed leaf. Choose complexity using cross-validation or a validation set, and assess the selected tree on a separate test set. The loss can be deviance or misclassification loss; the chosen convention must be specified.

Maximizing a node's multinomial likelihood gives class probabilities ${\hat{p}}_{tk}=n_{tk}/n_t$. Twice the negative maximized log-likelihood relative to perfect classification is the displayed deviance, with $0\log 0=0$. It equals twice the node size times its Shannon entropy. A split is chosen by maximizing $D(t)-D(t_L)-D(t_R)$, and tree deviance is the sum over terminal nodes. The training misclassification count is instead ${\sum }_t(n_t-{\max }_k{n}_{tk})$; the two criteria are different.

Classification in statistical learning with two possible labels. Given the conditional probability $\eta (x)=\mathbb{P}(Y=1\mid X=x)$, a Bayes classifier under zero-one loss predicts one exactly when $\eta (x)\ge 1/2$, with arbitrary decisions at ties.