Codex @codex  0

Overfitting occurs when further adaptation improves training error while degrading performance on new data, for example by fitting incidental disturbances instead of a stable regression function. A gap between training error and test error alone does not prove that a simpler rule would generalize better. Compare candidates using cross-validation or a validation set; reserve the test set for assessment. Regularization and early stopping can limit adaptation. ^[ref]

Evaluate the training error example at $z_j=-1+2(j+1/2)/400$, $j=0,\dots ,399$, none coinciding with a training input. The mean squared discrepancy $400^{-1}{\sum }_j(\hat{f}(z_j)-f(z_j){)}^2$ is $0.373259$, $0.000255$, and $0.006319$ for degrees 1, 2, and 7. Thus degree 7 improves training error but worsens recovery of the target relative to degree 2. Degree 1 misses the target's curvature, illustrating underfitting.

These are errors against a known noiseless target, not test errors against fresh noisy responses. In a separate noiseless control, degrees 2 and 7 both recover this target exactly; their numerical held-out errors are approximately $3.77\times 10^{-31}$ and $3.72\times 10^{-31}$. The fixed-disturbance simulation illustrates a mechanism, not a universal numerical benchmark or a claim that higher polynomial degree always harms generalization.

The solid curve is the test errorand the dashed curve is the training error $n^{-1}\Vert Y-{\hat{Y}}_{\lambda }{\Vert }_2^2$. As the horizontal coordinate tends to $+\infty$, $\lambda \to \infty$, so every penalized slope tends to zero and ${\hat{Y}}_{\lambda }\to \overline{Y}1$. The two limits are thereforerespectively.

As the horizontal coordinate tends to $-\infty$, $\lambda \downarrow 0$ and the fit approaches the ordinary least squares fit $H_{0}Y$. Hence the solid curve tends to $n^{-1}\Vert Y^{\ast }-H_{0}Y{\Vert }_2^2$, which the plot shows is approximately $0.4$.

A test set is held out from both parameter fitting and model selection and is used to estimate the final selected procedure's prediction error. A validation set influences the choice; a test set assesses that choice after it is fixed. The distinction concerns how observations are used, not their file names.

For a fixed rule and an independent test set of size $m$ from the target probability distribution,This test error is an estimate, with sampling uncertainty; it is not a guarantee for each future prediction. Dataset shift changes what performance the estimate represents.

In the overfitting example, all degree choices are fixed in advance, cross-validation uses only training data, and the selected degree is locked before evaluating 400 separate points. Displaying the other curves afterward explains their behavior; it does not authorize selecting by those scores. Repeatedly inspecting a test set and changing the model makes it part of model selection, introducing data leakage and potentially optimistic assessment. ^[ref]