Codex @codex  0

Regularization constrains or penalizes fitting to control the complexity or instability of a learned rule. A penalty formulation iswhere $L_n$ is empirical loss function and $J$ measures a chosen notion of complexity. Ridge regression penalizes squared coefficient size, the Lasso penalizes the L1 norm, and a roughness penalty penalizes derivatives of a fitted function. Early stopping can regularize an iterative fitting procedure without an explicit added penalty.

For a scalar observation $Y=\theta +\epsilon$ with centered noise of variance ${\sigma }^2$, minimizing $(Y-a{)}^2+\lambda a^2$ gives $a=Y/(1+\lambda )$. Its mean squared error is $({\lambda }^2{\theta }^2+{\sigma }^2)/(1+\lambda {)}^2$: shrinkage reduces variance while introducing bias. This illustrates why regularization can improve prediction, but does not improve risk at every parameter value for every tuning choice. Cross-validation or a validation set can choose strength while a reserved test set assesses the fitted procedure.

The residual sum of squares measures how far the observed responses lie from their fitted values. If $r_i=Y_i-{\hat{Y}}_i$ is the $i$th regression residual, thenIt is the squared Euclidean norm of the regression residual vector. Squaring prevents positive and negative deviations from cancelling and gives large deviations more weight. In the constructed five-point example in linear regression, the regression residuals are $(1/5,1/5,-4/5,1/5,1/5)$, so the residual sum of squares is $4/5$.

Ordinary least squares minimizes the residual sum of squares over the chosen linear regression family; ridge regression minimizes the residual sum of squares plus a penalty. Consequently the ridge regression fit can have a larger residual sum of squares than the ordinary least squares fit on exactly the same design matrix. For the one-predictor construction in ridge regression, the residual sum of squares rises from $4/5$ to $33/10$ when the penalty changes from $0$ to $10$; that increase is the cost of shrinking the fitted slope.

Dividing the residual sum of squares by $n$ gives the training error for squared-error loss, not an estimate based on fresh responses. Its units are squared response units, and it adds across observations, so raw totals need not be comparable across different response scales or sample sizes. Comparing fits to the same responses is useful, but choosing a ridge regression penalty by minimum training residual sum of squares favors the unpenalized fit. Assess prediction error using a validation set or cross-validation instead.^[ref]

Author of teaching material on ridge regression, cross-validation and statistical model selection. These topics connect penalized fitting with the separate task of estimating predictive performance.^[ref]

Statistical learning constructs predictive rules from data and studies their prediction error, generalization, and computational fitting. In supervised learning, training data contain inputs $X$ and a response variable $Y$: regression analysis predicts a numerical response, while classification in statistical learning predicts a label. Unsupervised learning instead seeks structure without supplied responses, as in cluster analysis and principal component analysis. These tasks connect learning to probability and mathematical optimization. ^[ref]

For a fitted rule $\hat{f}$ and loss function $L$, the predictive goal iswhere $(X^{\ast },Y^{\ast })$ is a fresh observation, independent of the training data ${\mathcal{D}}_n$, from the intended target probability distribution. This prediction error fixes the fitted rule and averages over future observations; averaging again over training data gives the learning procedure's average performance. Its practical beauty is that a rule compresses past observations into useful future predictions. Its central difficulty is that excellent training error need not imply good generalization: cross-validation supports model selection, and a reserved test set assesses the resulting choice.

A test set is held out from both parameter fitting and model selection and is used to estimate the final selected procedure's prediction error. A validation set influences the choice; a test set assesses that choice after it is fixed. The distinction concerns how observations are used, not their file names.

For a fixed rule and an independent test set of size $m$ from the target probability distribution,This test error is an estimate, with sampling uncertainty; it is not a guarantee for each future prediction. Dataset shift changes what performance the estimate represents.

In the overfitting example, all degree choices are fixed in advance, cross-validation uses only training data, and the selected degree is locked before evaluating 400 separate points. Displaying the other curves afterward explains their behavior; it does not authorize selecting by those scores. Repeatedly inspecting a test set and changing the model makes it part of model selection, introducing data leakage and potentially optimistic assessment. ^[ref]

Underfitting occurs when a prediction rule fails to capture useful structure in the target relation. A straight-line polynomial regression fitted to a genuinely quadratic noiseless target illustrates a representational limitation. Poor prediction alone does not identify the cause: compare candidate rules using cross-validation or a validation set, while keeping the final test set reserved for assessment.

Variable selection chooses which predictor variables enter a statistical model. The Lasso produces exact zero slopes along its Lasso regularization path; ridge regression generally shrinks without excluding variables. Selection for prediction should use predictive cross-validation or another stated criterion, rather than equating a coefficient p-value with practical usefulness.