= Test set
{title2=$\mathcal D_{\mathrm{test}}$}
A test set is held out from both parameter fitting and <model selection> and is used to estimate the final selected procedure's <prediction error>. A <validation set> influences the choice; a <test set> assesses that choice after it is fixed. The distinction concerns how observations are used, not their file names.
For a fixed rule and an <independent> <test set> of size $m$ from the target <probability distribution>,
$$
\widehat R_{\mathrm{test}}=\frac1m\sum_{j=1}^m L(Y_j^*,\widehat f(X_j^*)),\qquad
\mathbb E[\widehat R_{\mathrm{test}}\mid\mathcal D_n]=R(\widehat f).
$$
This <test error> is an estimate, with sampling uncertainty; it is not a guarantee for each future prediction. <Dataset shift> changes what performance the estimate represents.
In the <overfitting> example, all degree choices are fixed in advance, <cross-validation> uses only <training data>, and the selected degree is locked before evaluating 400 separate points. Displaying the other curves afterward explains their behavior; it does not authorize selecting by those scores. Repeatedly inspecting a <test set> and changing the model makes it part of <model selection>, introducing <data leakage> and potentially optimistic assessment. https://jmlr.org/papers/v11/cawley10a.html[Cawley and Talbot, over-fitting in model selection, 2010]{ref}
Back to article page
