= Solution
First use the dual convention for <support vectors>, and interpret the stipulated uniqueness as applying to each training fit. If $\alpha_j=0$, the <Karush-Kuhn-Tucker conditions> in the preceding part give $\xi_j=0$ and $y_jf(x_j)\geq1$. Delete the $j$th primal variable and its zero dual coefficient. All remaining primal constraints, stationarity equations and complementary-slackness equations still hold with the same $w,b$. These conditions certify a global minimizer of the reduced <convex optimization> problem, since the objective is convex and the <Slater condition> holds by choosing positive large slacks. Uniqueness therefore gives exactly the same fitted score and <support-vector-machine decision boundary>. The omitted point is correctly classified.
For the geometric convention, a non-support point has strict margin $y_jf(x_j)>1$. One can obtain the same conclusion using just uniqueness of the full fit: if the reduced problem had another optimizer, the convex segment from the original optimizer toward it would also consist of reduced optimizers. A sufficiently short segment retains strict margin at $j$, hence has zero omitted hinge loss and would give another full optimizer, a contradiction.
Only a <support vector> can therefore cause a held-out error. Each contributes at most one, proving the <support-vector leave-one-out error bound>
$$
\boxed{\operatorname{err}_{CV}\leq\frac{s}{n}.}
$$
Keeping $C$ fixed is necessary for this deletion argument: holding an average-loss penalty parameter fixed while changing the sample size changes the optimization problem's effective $C$.
Back to article page
