Training error (${\hat{R}}_{\mathrm{train}}=n^{-1}{\sum }_{i}L(Y_i,\hat{f}(X_i))$)

Training error is the average loss function evaluated on the observations used to fit the prediction rule. It is the empirical risk of the fitted rule on its own training data. With squared-error loss, it is the average squared regression residual, measuring fit to observed responses rather than performance on fresh responses. Flexible fitting usually makes it optimistically biased for prediction error, as quantified by prediction optimism.

For a reproducible polynomial regression example, take $x_i=-1+2i/11$, $i=0,\dots ,11$, and the known noiseless target $f(x)=1+x/2+2x^2$. Construct responses $y_i=f(x_i)+e_i$ with fixed disturbancesOrdinary least squares over polynomials of polynomial degree at most 1, 2, and 7 gives training errors $0.518937$, $0.015784$, and $0.009257$. The monotone improvement follows because these candidate families are nested: every smaller-degree fit remains available in the larger family. This argument concerns minimum training error, not generalization, and assumes the fitting mathematical optimization is solved. Degrees 2 and 7 fitted instead to the exact noiseless responses $f(x_i)$ have mathematically zero training error; the script obtains approximately $5.31\times 10^{-31}$ and $7.82\times 10^{-31}$ from rounding. Extra flexibility alone does not force overfitting.

Prediction optimism is the difference between expected squared prediction error on new independent responses at the training covariates and expected training error. Reusing responses to fit and assess a model tends to make training error too favorable. The covariance formula for prediction optimism measures this reuse without assuming a linear fitting rule.

// Target: probability-and-statistics.bigb

With fixed covariates and independent new responses having the same marginal distributions as the training responses, expanding squared losses cancels their second moments. Independence makes $\mathbb{E}(Y_i^{\mathrm{new}}{\hat{Y}}_i)=\mathbb{E}{Y}_i\mathbb{E}{\hat{Y}}_i$, yielding the displayed identity. For a fixed linear smoother $\hat{Y}=HY$ with covariance ${\sigma }^{2}I$, it reduces to $2{\sigma }^2\mathrm{tr}(H)/n$, linking prediction optimism to effective degrees of freedom.

// Target: analysis.bigb