Statistical learning

Statistical learning constructs predictive rules from data and studies their risk, generalization, and computational fitting.

Classification predicts a discrete label from observed covariates.

The L-nearest-neighbour classifier estimates each class probability by its empirical frequency among the $L$ nearest training covariates and predicts a class of greatest estimated probability.

For binary classification with posterior estimate ${\hat{p}}_1$, the plug-in classifier satisfies

A support vector machine chooses a separating decision function by trading a wide geometric margin against hinge-loss violations.

The slack variables satisfy ${\xi }_i\ge \max \{0,1-y_{i}f(x_i)\}$ and quantify margin violations.

For a linear score $f(x)=\alpha +x^T\beta$, the decision boundary is $f(x)=0$.

For a linear support vector machine, the geometric margin lies between $f(x)=-1$ and $f(x)=1$ and has width $2/\Vert \beta \Vert$.

The two margin boundaries of a linear support vector machine are the parallel hyperplanes $f(x)=1$ and $f(x)=-1$.

A support vector is a training observation on or inside the fitted margin; it has a nonzero dual coefficient and can affect the fitted boundary.

Cross-validation estimates predictive performance by repeatedly fitting on one part of the data and evaluating on held-out observations.

Leave-one-out cross-validation performs one fit with each observation held out in turn and averages the held-out losses.

K-fold cross-validation partitions observations into $K$ folds, trains $K$ times while holding out one fold at a time, and combines the held-out losses.

A neural network composes affine maps with nonlinear activation functions and learns their weights from data.

For logits $z_1,\dots ,z_K$, the softmax probabilities are $p_k=e^{z_k}/{\sum }_j{e}^{z_j}$.

For one-hot label $y$ and predicted class probabilities $p$, categorical cross-entropy is $-{\sum }_k{y}_k\log p_k$, the negative categorical log-likelihood.

Overfitting occurs when further adaptation improves training performance while degrading performance on new data.

Regularization restricts or penalizes model fitting to improve generalization.

Early stopping halts iterative fitting near the best validation performance, limiting adaptation to training noise.