 Statistical learning

Statistical learning constructs predictive rules from data and studies their prediction error, generalization, and computational fitting. In supervised learning, training data contain inputs $X$ and a response variable $Y$: regression analysis predicts a numerical response, while classification in statistical learning predicts a label. Unsupervised learning instead seeks structure without supplied responses, as in cluster analysis and principal component analysis. These tasks connect learning to probability and mathematical optimization. ^[ref]

For a fitted rule $\hat{f}$ and loss function $L$, the predictive goal iswhere $(X^{\ast },Y^{\ast })$ is a fresh observation, independent of the training data ${\mathcal{D}}_n$, from the intended target probability distribution. This prediction error fixes the fitted rule and averages over future observations; averaging again over training data gives the learning procedure's average performance. Its practical beauty is that a rule compresses past observations into useful future predictions. Its central difficulty is that excellent training error need not imply good generalization: cross-validation supports model selection, and a reserved test set assesses the resulting choice.