Classification in statistical learning

Classification predicts a discrete label from observed covariates.

The L-nearest-neighbour classifier estimates each class probability by its empirical frequency among the $L$ nearest training covariates and predicts a class of greatest estimated probability.

For binary classification with posterior estimate ${\hat{p}}_1$, the plug-in classifier satisfies

A support vector machine chooses a separating decision function by trading a wide geometric margin against hinge-loss violations.

The slack variables satisfy ${\xi }_i\ge \max \{0,1-y_{i}f(x_i)\}$ and quantify margin violations.

For a linear score $f(x)=\alpha +x^T\beta$, the decision boundary is $f(x)=0$.

For a linear support vector machine, the geometric margin lies between $f(x)=-1$ and $f(x)=1$ and has width $2/\Vert \beta \Vert$.

The two margin boundaries of a linear support vector machine are the parallel hyperplanes $f(x)=1$ and $f(x)=-1$.

A support vector is a training observation on or inside the fitted margin; it has a nonzero dual coefficient and can affect the fitted boundary.