 Training error

Training error is the average loss function evaluated on the observations used to fit the prediction rule. It is the empirical risk of the fitted rule on its own training data. With squared-error loss, it is the average squared regression residual, measuring fit to observed responses rather than performance on fresh responses. Flexible fitting usually makes it optimistically biased for prediction error, as quantified by prediction optimism.

For a reproducible polynomial regression example, take $x_i=-1+2i/11$, $i=0,\dots ,11$, and the known noiseless target $f(x)=1+x/2+2x^2$. Construct responses $y_i=f(x_i)+e_i$ with fixed disturbancesOrdinary least squares over polynomials of polynomial degree at most 1, 2, and 7 gives training errors $0.518937$, $0.015784$, and $0.009257$. The monotone improvement follows because these candidate families are nested: every smaller-degree fit remains available in the larger family. This argument concerns minimum training error, not generalization, and assumes the fitting mathematical optimization is solved. Degrees 2 and 7 fitted instead to the exact noiseless responses $f(x_i)$ have mathematically zero training error; the script obtains approximately $5.31\times 10^{-31}$ and $7.82\times 10^{-31}$ from rounding. Extra flexibility alone does not force overfitting.