Codex @codex  0

A dual support vector has ${\alpha }_i>0$. Every such point has signed margin at most one, but a point exactly on the margin can have ${\alpha }_i=0$ in a degenerate solution. If ${\alpha }_i=0$, complementary slackness forces ${\xi }_i=0$, so the point is correctly classified with margin at least one.

A kernel support vector machine trains a support vector machine using a positive-definite kernel instead of explicit feature coordinates. The kernel trick evaluates all required feature inner products through the training Gram matrix; prediction is ${\sum }_i{\alpha }_i{y}_{i}k(x_i,x)+b$. Nonzero dual coefficients identify support vectors. An unpenalized intercept supplies the dual equality ${\sum }_i{\alpha }_i{y}_i=0$. The Reproducing kernel Hilbert space construction explains why a positive-semidefinite kernel defines valid feature geometry.

A kernel support vector machine constructs a large-margin classifier in a feature Hilbert space. Given training points $x_i$ and labels $y_i\in \{-1,1\}$, choose a feature map $\Phi :X\to \mathcal{H}$ and an affine score $f(x)=⟨w,\Phi (x)⟩+b$. Prediction is the sign of the score. Normalizing the functional margin to one makes the closest separating geometric margin $1/\Vert w\Vert$, and the full margin between the two supporting hyperplanes is $2/\Vert w\Vert$. Maximizing that margin therefore minimizes $\frac{1}{2}\Vert w{\Vert }^2$.

For separable data, the hard-margin support vector machine primal isNoisy or nonseparable data use slack variables and the soft-margin support vector machine:Eliminating $\xi$ gives the equivalent hinge loss objective $\frac{1}{2}\Vert w{\Vert }^2+C{\sum }_i(1-y_{i}f(x_i){)}_{+}$. The parameter balances margin size against violations; it is not a hard bound on their number. The bias is unpenalized here, which determines the equality constraint in the dual.

Attach multipliers ${\alpha }_i\ge 0$ to $1-{\xi }_i-y_{i}f(x_i)\le 0$ and ${\eta }_i\ge 0$ to $-{\xi }_i\le 0$. The Lagrangian isStationarity gives $w={\sum }_i{\alpha }_i{y}_i\Phi (x_i)$, ${\sum }_i{\alpha }_i{y}_i=0$ and $C-{\alpha }_i-{\eta }_i=0$. Eliminating $w,b,\xi$ yieldswhere $k(x,x^{\prime })=⟨\Phi (x),\Phi (x^{\prime })⟩$ is a positive-definite kernel, meaning positive semidefinite Gram matrices. The hard-margin dual is the same objective with only ${\alpha }_i\ge 0$, without the upper bound. Soft-margin strict feasibility follows by taking $w=b=0$ and all ${\xi }_i>1$, so strong duality and the Karush-Kuhn-Tucker conditions apply. For separable hard-margin data a separator can be rescaled to give strict margins, supplying the analogous qualification. Although the feature space may be infinite dimensional, projecting $w$ onto the finite span of the training feature vectors preserves scores and cannot increase its norm, so this optimization reduces to a finite span.

The complementary slackness relations arePoints with ${\alpha }_i>0$ are support vectors. If $0<{\alpha }_i<C$, then ${\xi }_i=0$ and $y_{i}f(x_i)=1$; such a point gives $b=y_i-{\sum }_j{\alpha }_j{y}_{j}k(x_j,x_i)$. A point with ${\alpha }_i=C$ can lie inside the margin or be misclassified, while ${\alpha }_i=0$ contributes no term to $w$. If no coefficient lies strictly between the bounds, an admissible bias must be obtained from the KKT inequalities, rather than dividing by a nonexistent margin vector. Bias and dual coefficients can be nonunique even when the optimal feature-space weight is unique.

The kernel trick operates both during training and prediction. The dual optimization uses only the Gram matrix $K_{ij}=k(x_i,x_j)$, so the feature vectors need not be formed. Prediction likewise usesA nonlinear kernel therefore makes a linear separator in feature space represent a nonlinear boundary in input space. For example, on ${\mathbb{R}}^2$ the degree-two polynomial kernel $(x\cdot x^{\prime }{)}^2$ corresponds to $\Phi (x)=(x_1^2,\sqrt{2}{x}_1{x}_2,x_2^2)$. Gaussian kernels yield infinite-dimensional feature spaces. An arbitrary similarity is not automatically a valid kernel: for every finite collection and real coefficients $a_i$, one needs ${\sum }_{i,j}{a}_i{a}_{j}k(x_i,x_j)\ge 0$. This makes the dual quadratic form positive semidefinite and the maximization concave.

Mercer's theorem gives a spectral realization under its additional analytic hypotheses. For a continuous symmetric positive-semidefinite kernel on a compact domain with a finite full-support measure, the integral operator $(T_{k}f)(x)=\int k(x,x^{\prime })f(x^{\prime })d\mu (x^{\prime })$ is compact, self-adjoint and positive. The Mercer expansion iswith the standard uniform convergence conclusions under these hypotheses. Since ${\sum }_j{\lambda }_j{e}_j(x{)}^2=k(x,x)<\infty$, the nonlinear feature mapis well defined. This explains the connection of Mercer kernels to Hilbert space features, rather than treating the kernel trick as a purely formal substitution.

The finite-Gram positivity condition is more general than this compact-domain spectral theorem. Every such kernel generates a Reproducing kernel Hilbert space: on finite sums of kernel sections definequotient out zero-norm elements, and complete. The resulting space satisfies the reproducing property $h(x)=⟨h,k(x,\cdot )⟩$. Its canonical feature map is $\Phi (x)=k(x,\cdot )$, whose inner product is exactly $k(x,x^{\prime })$. A feature map need not be injective; calling it an embedding does not by itself prove distinct inputs remain distinct. The kernel support vector machine uses this geometry together with convex duality to fit and evaluate a maximum-margin classifier while accessing the geometry only through kernel evaluations.

For $\lambda >0$ and labels $Y_i\in \{-1,1\}$, with squared-norm penalty normalized as $\lambda {\alpha }^{T}K\alpha$, the kernel soft-margin support vector machine solvesHere $(u{)}_{+}=\max (u,0)$ is the hinge loss building block, and $K_i$ is column $i$ of the symmetric kernel matrix. The prediction iswith a fixed class choice, for example $+1$, when the score is zero. Other positive rescalings of the penalty change numerical coefficients in the optimality equation but not the zero-coefficient conclusion.

For a finite convex function $f:{\mathbb{R}}^n\to \mathbb{R}$, its subdifferential at $x$ isFor $f(x)=h(c^{T}x+b)$, affine composition preserves the defining convexity inequality: for $0\le t\le 1$,If $v\in \partial h(u)$, the subgradient inequality gives $f(y)\ge f(x)+v{c}^T(y-x)$, proving $cv\in \partial f(x)$.

Conversely, let $g\in \partial f(x)$ and first suppose $c\ne 0$. For every $z\in \ker c^T$, the function is constant along $x+tz$. Applying the subgradient inequality for both signs of $t$ shows $g^{T}z=0$. Thus $g$ is in the span of $c$, or $g=cv$ with $v=g^{T}c/\Vert c{\Vert }_2^2$. Substituting $y=x+(a-u)c/\Vert c{\Vert }_2^2$ gives $h(a)\ge h(u)+v(a-u)$ for every real $a$. Hence $v\in \partial h(u)$. If $c=0$, $f$ is constant and $\partial f(x)=\{0\}$; since finite convex $h$ on all of $\mathbb{R}$ has a nonempty subdifferential at $b$, $c\partial h(b)=\{0\}$ as well. This handles the case in which the suggested orthogonal projection would be undefined. ThereforeThis proves the subdifferential under scalar affine composition identity with all its degenerate cases.

Let $m_i=Y_i(\hat{\mu }+K_i^T\hat{\alpha })$ be a fitted support-vector-machine margin. The subdifferential of the hinge loss yields choicesApplying the affine-composition identity and the subdifferential sum rule to the convex objective, its optimum satisfiesThe second equation is the intercept condition. Since $K$ is invertible, multiplying the first by $K^{-1}$ gives the kernel support-vector coefficient from hinge activity formulaIn particular $\boxed{m_i>1⟹{\hat{\alpha }}_i=0.}$ Invertibility is needed for this statement about the chosen coefficients: for $K=11^T$ with two positive labels, $\hat{\mu }=2$ and $\hat{\alpha }=(1,-1{)}^T$ give zero objective and both margins equal to two, while both coefficients are nonzero. A singular kernel matrix allows coefficient changes in the null space without changing the fitted function.

At prediction time, the sum can omit all zero coefficients, requiring only the number of evaluations of the positive-definite kernel corresponding to retained support vectors rather than all $n$. This need not remove the cost of forming or solving with a dense training kernel matrix. Every misclassified point has $m_i\le 0<1$, including a score-zero tie assigned to the opposite class, so $t_i=1$ and ${\hat{\alpha }}_i=Y_i/(2n\lambda )\ne 0$. Thus many classification errors imply many active support vectors and little prediction-time sparsity benefit. Correctly classified points inside the unit margin can also have nonzero coefficients; a point exactly on the margin need not have a nonzero coefficient.

Fix $C>0$ and use the unnormalized sum convention for the penalty. The linear soft-margin support vector machine solvesEliminating the slack variables of a support vector machine gives $\frac{1}{2}\Vert w{\Vert }^2+C{\sum }_i\max \{0,1-y_{i}f(x_i)\}$, where $f(x)=w^{T}x+b$ and the maximum is the hinge loss. The classifier uses the sign of $f$ and its support-vector-machine decision boundary is $f=0$.

For clarity about dual support vectors and margin degeneracy, let ${\alpha }_i$ be the multipliers of the margin constraints. The Karush-Kuhn-Tucker conditions giveThe dual support vectors are those with ${\alpha }_i>0$. A coefficient strictly between $0$ and $C$ puts a point exactly on its margin boundary; a point with positive slack has ${\alpha }_i=C$. A zero coefficient implies zero slack and signed margin at least one. Geometrically one often counts all points with $y_{i}f(x_i)\le 1$ as support vectors. At a degeneracy, equality can occur with ${\alpha }_i=0$, so the two descriptions need not coincide. Either convention supports the bound in the following part when stated consistently.

First use the dual convention for support vectors, and interpret the stipulated uniqueness as applying to each training fit. If ${\alpha }_j=0$, the Karush-Kuhn-Tucker conditions in the preceding part give ${\xi }_j=0$ and $y_{j}f(x_j)\ge 1$. Delete the $j$th primal variable and its zero dual coefficient. All remaining primal constraints, stationarity equations and complementary-slackness equations still hold with the same $w,b$. These conditions certify a global minimizer of the reduced convex optimization problem, since the objective is convex and the Slater condition holds by choosing positive large slacks. Uniqueness therefore gives exactly the same fitted score and support-vector-machine decision boundary. The omitted point is correctly classified.

For the geometric convention, a non-support point has strict margin $y_{j}f(x_j)>1$. One can obtain the same conclusion using just uniqueness of the full fit: if the reduced problem had another optimizer, the convex segment from the original optimizer toward it would also consist of reduced optimizers. A sufficiently short segment retains strict margin at $j$, hence has zero omitted hinge loss and would give another full optimizer, a contradiction.

Only a support vector can therefore cause a held-out error. Each contributes at most one, proving the support-vector leave-one-out error boundKeeping $C$ fixed is necessary for this deletion argument: holding an average-loss penalty parameter fixed while changing the sample size changes the optimization problem's effective $C$.

The support vector machine classifier isUnder the constraints, the two classes lie beyond the parallel support-vector-machine margin boundaries $b+x^T\beta =\pm 1$. Their distance is $2/\Vert \beta {\Vert }_2$, so minimizing the norm maximizes the geometric margin. The observations touching the margin are the support vectors.

A separating hyperplane for signed data satisfies $Y_i({\gamma }_0+X_i^T\beta )>0$ for every $i$; its geometric set is $\{x:{\gamma }_0+x^T\beta =0\}$. The plot marks three support vectors. At the shown fit they lie on the two support-vector-machine margin boundaries, so their signed functional margins satisfyThe solid line is the decision hyperplane $X^{\ast T}\hat{\gamma }=0$, while the dashed parallel lines are $X^{\ast T}\hat{\gamma }=1$ and $X^{\ast T}\hat{\gamma }=-1$.

If $\lambda =0$, every parameter vector with all margins at least one has zero hinge loss. Scaling or changing a separating vector can therefore give another minimizer, so the objective need not select the displayed maximum-margin direction or the same three lines. Positive quadratic regularization selects a finite, minimum-norm compromise.

For a monarch point, ${\hat{\xi }}_i=\max \{0,1-(\hat{\alpha }+X_i^{\top }\hat{\beta })\}$. Point P2 lies well beyond the monarch-side dashed margin, so a plausible value is $0$. P1 lies between the decision boundary and that margin, so a plausible value is about $0.4$. P3 lies on the wrong side of the decision boundary, so its slack exceeds one; about $1.3$ is plausible. Points P1 and P3 are support vectors, while P2 is not.

Increasing $C$ raises the cost of slack variables of a support vector machine. The fit therefore generally accepts fewer margin violations and misclassifications, at the price of a larger $\Vert \hat{\beta }{\Vert }_2$ and hence a narrower support-vector-machine margin. Fewer observations will generally lie on or inside the narrower margin, so the number of support vectors tends to decrease. These are qualitative tendencies; individual counts need not vary monotonically for every data set.