Soft-margin support vector machine

For signed training labels, a linear soft-margin support vector machine minimizeswith $C>0$. Its Karush-Kuhn-Tucker conditions give $w={\sum }_i{\alpha }_i{y}_i{x}_i$, ${\sum }_i{\alpha }_i{y}_i=0$, and $0\le {\alpha }_i\le C$.

For the kernel support vector machine objective $n^{-1}{\sum }_i(1-Y_i(\mu +K_i^T\alpha ){)}_{+}+\lambda {\alpha }^{T}K\alpha$, assume $Y_i\in \{-1,1\}$, $\lambda >0$, and that the kernel matrix $K$ is invertible. The subdifferential optimality equation givesThus strict margins greater than one force zero coefficients, while misclassified observations have nonzero coefficients. Invertibility matters: a singular kernel matrix allows coefficient changes in its null space without changing the fitted function or objective.

For a fixed $C$ in the unnormalized sum-of-slacks objective and unique training optimizers, deleting an observation with ${\alpha }_i=0$ preserves the Karush-Kuhn-Tucker conditions and the fitted decision function. It is correctly classified when held out. Consequently Leave-one-out cross-validation makes at most $s$ errors, where $s$ counts positive dual coefficients. With geometric support vectors, counting all points of signed margin at most one also gives the bound, without a nondegeneracy assumption.

A dual support vector has ${\alpha }_i>0$. Every such point has signed margin at most one, but a point exactly on the margin can have ${\alpha }_i=0$ in a degenerate solution. If ${\alpha }_i=0$, complementary slackness forces ${\xi }_i=0$, so the point is correctly classified with margin at least one.