= Overfitting
{wiki}
Overfitting occurs when further adaptation improves <training error> while degrading performance on new data, for example by fitting incidental disturbances instead of a stable <regression function>. A gap between <training error> and <test error> alone does not prove that a simpler rule would generalize better. Compare candidates using <cross-validation> or a <validation set>; reserve the <test set> for assessment. <Regularization> and <early stopping> can limit adaptation. https://web.stanford.edu/~hastie/ISLR2/Slides/Ch2_Statistical_Learning.pdf[ISLR chapter 2 slides, assessing model accuracy]{ref}
Evaluate the <training error> example at $z_j=-1+2(j+1/2)/400$, $j=0,\ldots,399$, none coinciding with a training input. The mean squared discrepancy $400^{-1}\sum_j(\widehat f(z_j)-f(z_j))^2$ is $0.373259$, $0.000255$, and $0.006319$ for degrees 1, 2, and 7. Thus degree 7 improves <training error> but worsens recovery of the target relative to degree 2. Degree 1 misses the target's curvature, illustrating <underfitting>.
These are errors against a known noiseless target, not <test errors> against fresh noisy responses. In a separate noiseless control, degrees 2 and 7 both recover this target exactly; their numerical held-out errors are approximately $3.77\times10^{-31}$ and $3.72\times10^{-31}$. The fixed-disturbance simulation illustrates a mechanism, not a universal numerical benchmark or a claim that higher <polynomial degree> always harms <generalization>.
\Image[/learning-fit-vs-generalization.png]
{title=Noisy polynomial fitting, training-only degree selection, and separate noiseless-target evaluation}
Back to article page
