= Training error
{title2=$\widehat R_{\mathrm{train}}=n^{-1}\sum_i L(Y_i,\widehat f(X_i))$}
= In-sample error
{synonym}
Training error is the average <loss function> evaluated on the observations used to fit the prediction rule. It is the <empirical risk> of the fitted rule on its own <training data>. With <squared-error loss>, it is the average squared <regression residual>, measuring fit to observed responses rather than performance on fresh responses. Flexible fitting usually makes it optimistically biased for <prediction error>, as quantified by <prediction optimism>.
For a reproducible <polynomial regression> example, take $x_i=-1+2i/11$, $i=0,\ldots,11$, and the known noiseless target $f(x)=1+x/2+2x^2$. Construct responses $y_i=f(x_i)+e_i$ with fixed disturbances
$$
(e_i)=(0.12,-0.17,0.10,0.18,-0.13,0.08,-0.16,0.11,-0.08,0.15,-0.12,0.06).
$$
<Ordinary least squares> over <polynomials> of <polynomial degree> at most 1, 2, and 7 gives <training errors> $0.518937$, $0.015784$, and $0.009257$. The monotone improvement follows because these candidate families are nested: every smaller-degree fit remains available in the larger family. This argument concerns minimum <training error>, not <generalization>, and assumes the fitting <mathematical optimization> is solved. Degrees 2 and 7 fitted instead to the exact noiseless responses $f(x_i)$ have mathematically zero <training error>; the script obtains approximately $5.31\times10^{-31}$ and $7.82\times10^{-31}$ from rounding. Extra flexibility alone does not force <overfitting>.
Back to article page
