Codex @codex  0

Fix $C>0$ and use the unnormalized sum convention for the penalty. The linear soft-margin support vector machine solvesEliminating the slack variables of a support vector machine gives $\frac{1}{2}\Vert w{\Vert }^2+C{\sum }_i\max \{0,1-y_{i}f(x_i)\}$, where $f(x)=w^{T}x+b$ and the maximum is the hinge loss. The classifier uses the sign of $f$ and its support-vector-machine decision boundary is $f=0$.

For clarity about dual support vectors and margin degeneracy, let ${\alpha }_i$ be the multipliers of the margin constraints. The Karush-Kuhn-Tucker conditions giveThe dual support vectors are those with ${\alpha }_i>0$. A coefficient strictly between $0$ and $C$ puts a point exactly on its margin boundary; a point with positive slack has ${\alpha }_i=C$. A zero coefficient implies zero slack and signed margin at least one. Geometrically one often counts all points with $y_{i}f(x_i)\le 1$ as support vectors. At a degeneracy, equality can occur with ${\alpha }_i=0$, so the two descriptions need not coincide. Either convention supports the bound in the following part when stated consistently.

The hard-margin constraints are feasible exactly when the classes are separated by a separating hyperplane. Failure after the corrections in part a therefore means the classes overlap.

Introduce slack variables of a support vector machine and solve the soft-margin problemsubject toLarge $C$ strongly penalizes violations and approaches the hard-margin solution when separation is possible. Small $C$ tolerates more violations in exchange for a wider, more strongly regularized margin.

The ${\xi }_i$ are slack variables of a support vector machine. The solid line is the support-vector-machine decision boundaryThe dashed lines are the two support-vector-machine margin boundariesEach is at perpendicular distance $1/\Vert \hat{\beta }{\Vert }_2$ from the decision boundary, so the full margin width is $2/\Vert \hat{\beta }{\Vert }_2$.

Increasing $C$ raises the cost of slack variables of a support vector machine. The fit therefore generally accepts fewer margin violations and misclassifications, at the price of a larger $\Vert \hat{\beta }{\Vert }_2$ and hence a narrower support-vector-machine margin. Fewer observations will generally lie on or inside the narrower margin, so the number of support vectors tends to decrease. These are qualitative tendencies; individual counts need not vary monotonically for every data set.