Cross-validation ($\mathrm{CV}=n^{-1}{\sum }_{i}L(Y_i,{\hat{f}}^{(-k(i))}(X_i))$)

Cross-validation estimates predictive performance by repeatedly fitting on one part of the data and evaluating on held-out observations. In K-fold cross-validation, divide training data into disjoint folds $I_1,\dots ,I_K$; $k(i)$ identifies observation $i$'s fold, and ${\hat{f}}^{(-k)}$ is fitted without fold $I_k$. The displayed average scores each observation using a rule that did not train on it. Weight by observation count when fold sizes differ. Leave-one-out cross-validation uses one observation per fold. ^[ref]

For the training error example, three folds contain indices $i$ with equal remainder modulo 3. Each fit uses eight observations. The squared-error loss averages for degrees 1, 2, and 7 are $0.724132$, $0.027192$, and $1.532790$ respectively. Select degree 2 and refit on all twelve training data observations. The folds reuse scarce data while challenging each candidate to predict unseen responses; they do not touch the final test set.

Any learned preprocessing, including variable selection, must also be fitted within each training fold to avoid data leakage. ^[ref] Repeated tuning can itself overfit the cross-validation score; nested cross-validation uses inner folds for model selection and outer folds for assessment of that entire procedure. ^[ref]

Folds use fewer training observations than the final refit, and their scores are dependent because training observations overlap. ^[ref] Splitting must match deployment: for time series forecasting, rolling-origin cross-validation trains on the past and evaluates later observations, preventing future information from entering a forecast. Ordinary random folds need not answer that forecasting question. ^[ref]

Nested cross-validation separates selection from assessment using two levels of held-out data. Within each outer training fold, inner cross-validation chooses a model or tuning parameters; the complete selected fitting procedure is then applied to that outer training fold and scored on its untouched outer evaluation fold. This estimates the performance of the model selection procedure, including its choices, rather than reusing its best inner score as an assessment. A final model can then be selected and fitted using all available training data. ^[ref]

Subtract the diagonal variance contribution from the derivative estimate's squared norm, then insert the result into asymptotic mean integrated squared error. The expected curvature estimate equals $R(f^{\prime \prime })+O(h^2)$, apart from a negligible finite-n term. This makes the resulting criterion asymptotically correct even though it is not an exactly unbiased estimate of integrated risk. Under standard conditions its selector is ratio-consistent with the optimal bandwidth and has relative fluctuation order $n^{-1/10}$.

Minimize the displayed criterion using a Leave-one-out cross-validation estimate in the second term. Its expectation is the exact mean integrated squared error minus $R(f)$, which does not depend on h. With standard smoothness and appropriate search ranges, the chosen bandwidth is asymptotically optimal. Its relative fluctuations are ordinarily of order $n^{-1/10}$, considerably larger than the estimation error achievable by well-chosen plug-in pilots.

For a fixed linear smoother, generalized cross-validation replaces the individual leverage corrections of Leave-one-out cross-validation by their average. It selects smoothing parameters by minimizing the displayed criterion. It approximates a predictive error criterion; it is not a universal exact estimate of future error and requires care when observation errors are dependent.

Leave-one-out cross-validation performs one fit with each observation held out in turn and averages the held-out losses.

If a linear smoother has fitted vector $\hat{Y}=HY$, its residual after fitting without observation $i$ isThe identity follows from a block matrix inverse or a rank-one inverse update.

K-fold cross-validation partitions observations into $K$ folds, trains $K$ times while holding out one fold at a time, and combines the held-out losses.