Cross-validation
= Cross-validation
{title2=$\operatorname{CV}=n^{-1}\sum_i L(Y_i,\widehat f^{(-k(i))}(X_i))$}
{wiki=Cross-validation_(statistics)}
Cross-validation estimates predictive performance by repeatedly fitting on one part of the data and evaluating on held-out observations. In <K-fold cross-validation>, divide <training data> into disjoint folds $I_1,\ldots,I_K$; $k(i)$ identifies observation $i$'s fold, and $\widehat f^{(-k)}$ is fitted without fold $I_k$. The displayed average scores each observation using a rule that did not train on it. Weight by observation count when fold sizes differ. <Leave-one-out cross-validation> uses one observation per fold. https://web.stanford.edu/~hastie/ISLR2/Slides/Ch5_Resampling_Methods.pdf[ISLR chapter 5 slides, resampling methods]{ref}
For the <training error> example, three folds contain indices $i$ with equal remainder modulo 3. Each fit uses eight observations. The <squared-error loss> averages for degrees 1, 2, and 7 are $0.724132$, $0.027192$, and $1.532790$ respectively. Select degree 2 and refit on all twelve <training data> observations. The folds reuse scarce data while challenging each candidate to predict unseen responses; they do not touch the final <test set>.
Any learned preprocessing, including variable selection, must also be fitted within each training fold to avoid <data leakage>. https://web.stanford.edu/~hastie/ISLR2/Slides/Ch5_Resampling_Methods.pdf[ISLR chapter 5 slides, the wrong and right way]{ref} Repeated tuning can itself overfit the <cross-validation> score; <nested cross-validation> uses inner folds for <model selection> and outer folds for assessment of that entire procedure. <On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation by Gavin C. Cawley and others>{ref}
Folds use fewer training observations than the final refit, and their scores are dependent because training observations overlap. https://web.stanford.edu/~hastie/ISLR2/Slides/Ch5_Resampling_Methods.pdf[ISLR chapter 5 slides, correlated fold estimates]{ref} Splitting must match deployment: for <time series> forecasting, rolling-origin <cross-validation> trains on the past and evaluates later observations, preventing future information from entering a forecast. Ordinary random folds need not answer that forecasting question. https://otexts.com/fpp3/tscv.html[Hyndman and Athanasopoulos, time series cross-validation]{ref}