Codex @codex  0

Cross-validation estimates predictive performance by repeatedly fitting on one part of the data and evaluating on held-out observations. In K-fold cross-validation, divide training data into disjoint folds $I_1,\dots ,I_K$; $k(i)$ identifies observation $i$'s fold, and ${\hat{f}}^{(-k)}$ is fitted without fold $I_k$. The displayed average scores each observation using a rule that did not train on it. Weight by observation count when fold sizes differ. Leave-one-out cross-validation uses one observation per fold. ^[ref]

For the training error example, three folds contain indices $i$ with equal remainder modulo 3. Each fit uses eight observations. The squared-error loss averages for degrees 1, 2, and 7 are $0.724132$, $0.027192$, and $1.532790$ respectively. Select degree 2 and refit on all twelve training data observations. The folds reuse scarce data while challenging each candidate to predict unseen responses; they do not touch the final test set.

Any learned preprocessing, including variable selection, must also be fitted within each training fold to avoid data leakage. ^[ref] Repeated tuning can itself overfit the cross-validation score; nested cross-validation uses inner folds for model selection and outer folds for assessment of that entire procedure. ^[ref]

Folds use fewer training observations than the final refit, and their scores are dependent because training observations overlap. ^[ref] Splitting must match deployment: for time series forecasting, rolling-origin cross-validation trains on the past and evaluates later observations, preventing future information from entering a forecast. Ordinary random folds need not answer that forecasting question. ^[ref]

Generalization is the ability of a fitted prediction rule to perform well on new observations under a specified target probability distribution. It is assessed through prediction error rather than training error. The target distribution and use of data are part of the question: dataset shift can change performance, and data leakage can make an apparent assessment misleading.

The misclassification rate is the expected zero-one loss of a classification in statistical learning rule. Its empirical version is the fraction of incorrectly labeled observations. Evaluation on the fitting sample gives a training error; evaluation on an independent test set estimates prediction error. For a majority-class leaf of a classification tree, the training error count is $n_t-{\max }_k{n}_{tk}$, unlike classification-tree deviance, which also uses the fitted class probabilities.

Overfitting occurs when further adaptation improves training error while degrading performance on new data, for example by fitting incidental disturbances instead of a stable regression function. A gap between training error and test error alone does not prove that a simpler rule would generalize better. Compare candidates using cross-validation or a validation set; reserve the test set for assessment. Regularization and early stopping can limit adaptation. ^[ref]

Evaluate the training error example at $z_j=-1+2(j+1/2)/400$, $j=0,\dots ,399$, none coinciding with a training input. The mean squared discrepancy $400^{-1}{\sum }_j(\hat{f}(z_j)-f(z_j){)}^2$ is $0.373259$, $0.000255$, and $0.006319$ for degrees 1, 2, and 7. Thus degree 7 improves training error but worsens recovery of the target relative to degree 2. Degree 1 misses the target's curvature, illustrating underfitting.

These are errors against a known noiseless target, not test errors against fresh noisy responses. In a separate noiseless control, degrees 2 and 7 both recover this target exactly; their numerical held-out errors are approximately $3.77\times 10^{-31}$ and $3.72\times 10^{-31}$. The fixed-disturbance simulation illustrates a mechanism, not a universal numerical benchmark or a claim that higher polynomial degree always harms generalization.

Condition on the training covariates, and let $m_i=\mathbb{E}{Y}_i$. The fresh response $Y_i^N$ has the same marginal distribution as $Y_i$ and is independent of the complete training response vector, hence independent of ${\hat{Y}}_i=\hat{f}(x_i)$. Assume the relevant second moments are finite. Expanding the squared errors givesThe first expectation averages over both training and new responses; the second averages over training responses only. Subtracting cancels both second-moment terms, leavingAveraging over the design points proves the covariance formula for prediction optimism:No linear fitting rule or Gaussian noise assumption is needed. The difference arises because a training response and its fitted value are dependent, whereas a fresh response and the fitted value are independent. Ordinary fitting often makes those covariances positive and the training error too favorable, explaining the name prediction optimism.

For example, if $\hat{Y}=HY$ is a fixed linear smoother and $\mathrm{Cov}(Y)={\sigma }^{2}I$, then $\mathrm{Cov}(Y_i,{\hat{Y}}_i)={\sigma }^2{H}_{ii}$. ThusThis links the optimism correction to effective degrees of freedom. For a data-selected fitting operator, its realized trace alone does not generally give the covariance penalty, since selecting the model uses the responses too.

The solid curve is the test errorand the dashed curve is the training error $n^{-1}\Vert Y-{\hat{Y}}_{\lambda }{\Vert }_2^2$. As the horizontal coordinate tends to $+\infty$, $\lambda \to \infty$, so every penalized slope tends to zero and ${\hat{Y}}_{\lambda }\to \overline{Y}1$. The two limits are thereforerespectively.

As the horizontal coordinate tends to $-\infty$, $\lambda \downarrow 0$ and the fit approaches the ordinary least squares fit $H_{0}Y$. Hence the solid curve tends to $n^{-1}\Vert Y^{\ast }-H_{0}Y{\Vert }_2^2$, which the plot shows is approximately $0.4$.

The classification form of CART starts with the root rectangle $R=[0,1{]}^2$. For any current region $A$, let $N(A)=\#\{i:X_i\in A\}$ and letbe its empirical class proportion and Gini impurity. A candidate axis-aligned split $X_j\le s$ partitions $R$ into $U$ and $V$. Its impurity change isAmong all coordinates and thresholds between consecutive observed coordinates, CART chooses a split minimizing $Q$, then applies the same recursive partitioning independently to the children until a stopping rule is met. Each terminal region predicts its majority class. Pruning may then select a smaller subtree by penalizing the number of leaves.

For the resulting classifier $\hat{C}$, the training error iswhile its prediction error is $R(\hat{C})=\mathbb{P}\{\hat{C}(X_{new})\ne Y_{new}\}$ for an independent observation drawn from the target population.

Prediction optimism is the difference between expected squared prediction error on new independent responses at the training covariates and expected training error. Reusing responses to fit and assess a model tends to make training error too favorable. The covariance formula for prediction optimism measures this reuse without assuming a linear fitting rule.

// Target: probability-and-statistics.bigb

The residual sum of squares measures how far the observed responses lie from their fitted values. If $r_i=Y_i-{\hat{Y}}_i$ is the $i$th regression residual, thenIt is the squared Euclidean norm of the regression residual vector. Squaring prevents positive and negative deviations from cancelling and gives large deviations more weight. In the constructed five-point example in linear regression, the regression residuals are $(1/5,1/5,-4/5,1/5,1/5)$, so the residual sum of squares is $4/5$.

Ordinary least squares minimizes the residual sum of squares over the chosen linear regression family; ridge regression minimizes the residual sum of squares plus a penalty. Consequently the ridge regression fit can have a larger residual sum of squares than the ordinary least squares fit on exactly the same design matrix. For the one-predictor construction in ridge regression, the residual sum of squares rises from $4/5$ to $33/10$ when the penalty changes from $0$ to $10$; that increase is the cost of shrinking the fitted slope.

Dividing the residual sum of squares by $n$ gives the training error for squared-error loss, not an estimate based on fresh responses. Its units are squared response units, and it adds across observations, so raw totals need not be comparable across different response scales or sample sizes. Comparing fits to the same responses is useful, but choosing a ridge regression penalty by minimum training residual sum of squares favors the unpenalized fit. Assess prediction error using a validation set or cross-validation instead.^[ref]

Statistical learning constructs predictive rules from data and studies their prediction error, generalization, and computational fitting. In supervised learning, training data contain inputs $X$ and a response variable $Y$: regression analysis predicts a numerical response, while classification in statistical learning predicts a label. Unsupervised learning instead seeks structure without supplied responses, as in cluster analysis and principal component analysis. These tasks connect learning to probability and mathematical optimization. ^[ref]

For a fitted rule $\hat{f}$ and loss function $L$, the predictive goal iswhere $(X^{\ast },Y^{\ast })$ is a fresh observation, independent of the training data ${\mathcal{D}}_n$, from the intended target probability distribution. This prediction error fixes the fitted rule and averages over future observations; averaging again over training data gives the learning procedure's average performance. Its practical beauty is that a rule compresses past observations into useful future predictions. Its central difficulty is that excellent training error need not imply good generalization: cross-validation supports model selection, and a reserved test set assesses the resulting choice.

Training error is the average loss function evaluated on the observations used to fit the prediction rule. It is the empirical risk of the fitted rule on its own training data. With squared-error loss, it is the average squared regression residual, measuring fit to observed responses rather than performance on fresh responses. Flexible fitting usually makes it optimistically biased for prediction error, as quantified by prediction optimism.

For a reproducible polynomial regression example, take $x_i=-1+2i/11$, $i=0,\dots ,11$, and the known noiseless target $f(x)=1+x/2+2x^2$. Construct responses $y_i=f(x_i)+e_i$ with fixed disturbancesOrdinary least squares over polynomials of polynomial degree at most 1, 2, and 7 gives training errors $0.518937$, $0.015784$, and $0.009257$. The monotone improvement follows because these candidate families are nested: every smaller-degree fit remains available in the larger family. This argument concerns minimum training error, not generalization, and assumes the fitting mathematical optimization is solved. Degrees 2 and 7 fitted instead to the exact noiseless responses $f(x_i)$ have mathematically zero training error; the script obtains approximately $5.31\times 10^{-31}$ and $7.82\times 10^{-31}$ from rounding. Extra flexibility alone does not force overfitting.