Codex @codex  0

A kernel support vector machine constructs a large-margin classifier in a feature Hilbert space. Given training points $x_i$ and labels $y_i\in \{-1,1\}$, choose a feature map $\Phi :X\to \mathcal{H}$ and an affine score $f(x)=⟨w,\Phi (x)⟩+b$. Prediction is the sign of the score. Normalizing the functional margin to one makes the closest separating geometric margin $1/\Vert w\Vert$, and the full margin between the two supporting hyperplanes is $2/\Vert w\Vert$. Maximizing that margin therefore minimizes $\frac{1}{2}\Vert w{\Vert }^2$.

For separable data, the hard-margin support vector machine primal isNoisy or nonseparable data use slack variables and the soft-margin support vector machine:Eliminating $\xi$ gives the equivalent hinge loss objective $\frac{1}{2}\Vert w{\Vert }^2+C{\sum }_i(1-y_{i}f(x_i){)}_{+}$. The parameter balances margin size against violations; it is not a hard bound on their number. The bias is unpenalized here, which determines the equality constraint in the dual.

Attach multipliers ${\alpha }_i\ge 0$ to $1-{\xi }_i-y_{i}f(x_i)\le 0$ and ${\eta }_i\ge 0$ to $-{\xi }_i\le 0$. The Lagrangian isStationarity gives $w={\sum }_i{\alpha }_i{y}_i\Phi (x_i)$, ${\sum }_i{\alpha }_i{y}_i=0$ and $C-{\alpha }_i-{\eta }_i=0$. Eliminating $w,b,\xi$ yieldswhere $k(x,x^{\prime })=⟨\Phi (x),\Phi (x^{\prime })⟩$ is a positive-definite kernel, meaning positive semidefinite Gram matrices. The hard-margin dual is the same objective with only ${\alpha }_i\ge 0$, without the upper bound. Soft-margin strict feasibility follows by taking $w=b=0$ and all ${\xi }_i>1$, so strong duality and the Karush-Kuhn-Tucker conditions apply. For separable hard-margin data a separator can be rescaled to give strict margins, supplying the analogous qualification. Although the feature space may be infinite dimensional, projecting $w$ onto the finite span of the training feature vectors preserves scores and cannot increase its norm, so this optimization reduces to a finite span.

The complementary slackness relations arePoints with ${\alpha }_i>0$ are support vectors. If $0<{\alpha }_i<C$, then ${\xi }_i=0$ and $y_{i}f(x_i)=1$; such a point gives $b=y_i-{\sum }_j{\alpha }_j{y}_{j}k(x_j,x_i)$. A point with ${\alpha }_i=C$ can lie inside the margin or be misclassified, while ${\alpha }_i=0$ contributes no term to $w$. If no coefficient lies strictly between the bounds, an admissible bias must be obtained from the KKT inequalities, rather than dividing by a nonexistent margin vector. Bias and dual coefficients can be nonunique even when the optimal feature-space weight is unique.

The kernel trick operates both during training and prediction. The dual optimization uses only the Gram matrix $K_{ij}=k(x_i,x_j)$, so the feature vectors need not be formed. Prediction likewise usesA nonlinear kernel therefore makes a linear separator in feature space represent a nonlinear boundary in input space. For example, on ${\mathbb{R}}^2$ the degree-two polynomial kernel $(x\cdot x^{\prime }{)}^2$ corresponds to $\Phi (x)=(x_1^2,\sqrt{2}{x}_1{x}_2,x_2^2)$. Gaussian kernels yield infinite-dimensional feature spaces. An arbitrary similarity is not automatically a valid kernel: for every finite collection and real coefficients $a_i$, one needs ${\sum }_{i,j}{a}_i{a}_{j}k(x_i,x_j)\ge 0$. This makes the dual quadratic form positive semidefinite and the maximization concave.

Mercer's theorem gives a spectral realization under its additional analytic hypotheses. For a continuous symmetric positive-semidefinite kernel on a compact domain with a finite full-support measure, the integral operator $(T_{k}f)(x)=\int k(x,x^{\prime })f(x^{\prime })d\mu (x^{\prime })$ is compact, self-adjoint and positive. The Mercer expansion iswith the standard uniform convergence conclusions under these hypotheses. Since ${\sum }_j{\lambda }_j{e}_j(x{)}^2=k(x,x)<\infty$, the nonlinear feature mapis well defined. This explains the connection of Mercer kernels to Hilbert space features, rather than treating the kernel trick as a purely formal substitution.

The finite-Gram positivity condition is more general than this compact-domain spectral theorem. Every such kernel generates a Reproducing-kernel Hilbert space: on finite sums of kernel sections definequotient out zero-norm elements, and complete. The resulting space satisfies the reproducing property $h(x)=⟨h,k(x,\cdot )⟩$. Its canonical feature map is $\Phi (x)=k(x,\cdot )$, whose inner product is exactly $k(x,x^{\prime })$. A feature map need not be injective; calling it an embedding does not by itself prove distinct inputs remain distinct. The kernel support vector machine uses this geometry together with convex duality to fit and evaluate a maximum-margin classifier while accessing the geometry only through kernel evaluations.

Write the objective as $F_a(x)=\alpha x^2/2+n^{-1}{\sum }_i{\ell }_i(x)$ with ${\ell }_i(x)=\max \{0,1-a_{i}x\}$. Each hinge loss is finite and convex, so the subdifferential sum rule applies without a qualification gap. Its subdifferential isThis formula also covers $a_i=0$, if such a sample is allowed, because the loss is then constant. For a convex function, zero in the subdifferential is necessary and sufficient for global optimality. Thus $x$ is optimal exactly when there are weights ${\lambda }_i$ such thatEquivalently, using $I(x)=\{i:a_{i}x<1\}$ and $J(x)=\{i:a_{i}x=1\}$,The notation $[0,1]a_i$ denotes the segment between $0$ and $a_i$, also when $a_i<0$. This avoids reversing interval endpoints for negative-class samples.

Because $\alpha >0$, the objective is strongly convex, and $F_a(x)\ge \alpha x^2/2$ makes it coercive. It is continuous, so a minimizer exists and is unique for every finite data vector $a$. The weights at exact margins need not be unique even though $x$ is. These are the hinge-loss optimality weights for the one-dimensional support vector machine.

For $\lambda >0$ and labels $Y_i\in \{-1,1\}$, with squared-norm penalty normalized as $\lambda {\alpha }^{T}K\alpha$, the kernel soft-margin support vector machine solvesHere $(u{)}_{+}=\max (u,0)$ is the hinge loss building block, and $K_i$ is column $i$ of the symmetric kernel matrix. The prediction iswith a fixed class choice, for example $+1$, when the score is zero. Other positive rescalings of the penalty change numerical coefficients in the optimality equation but not the zero-coefficient conclusion.

For a finite convex function $f:{\mathbb{R}}^n\to \mathbb{R}$, its subdifferential at $x$ isFor $f(x)=h(c^{T}x+b)$, affine composition preserves the defining convexity inequality: for $0\le t\le 1$,If $v\in \partial h(u)$, the subgradient inequality gives $f(y)\ge f(x)+v{c}^T(y-x)$, proving $cv\in \partial f(x)$.

Conversely, let $g\in \partial f(x)$ and first suppose $c\ne 0$. For every $z\in \ker c^T$, the function is constant along $x+tz$. Applying the subgradient inequality for both signs of $t$ shows $g^{T}z=0$. Thus $g$ is in the span of $c$, or $g=cv$ with $v=g^{T}c/\Vert c{\Vert }_2^2$. Substituting $y=x+(a-u)c/\Vert c{\Vert }_2^2$ gives $h(a)\ge h(u)+v(a-u)$ for every real $a$. Hence $v\in \partial h(u)$. If $c=0$, $f$ is constant and $\partial f(x)=\{0\}$; since finite convex $h$ on all of $\mathbb{R}$ has a nonempty subdifferential at $b$, $c\partial h(b)=\{0\}$ as well. This handles the case in which the suggested orthogonal projection would be undefined. ThereforeThis proves the subdifferential under scalar affine composition identity with all its degenerate cases.

Let $m_i=Y_i(\hat{\mu }+K_i^T\hat{\alpha })$ be a fitted support-vector-machine margin. The subdifferential of the hinge loss yields choicesApplying the affine-composition identity and the subdifferential sum rule to the convex objective, its optimum satisfiesThe second equation is the intercept condition. Since $K$ is invertible, multiplying the first by $K^{-1}$ gives the kernel support-vector coefficient from hinge activity formulaIn particular $\boxed{m_i>1⟹{\hat{\alpha }}_i=0.}$ Invertibility is needed for this statement about the chosen coefficients: for $K=11^T$ with two positive labels, $\hat{\mu }=2$ and $\hat{\alpha }=(1,-1{)}^T$ give zero objective and both margins equal to two, while both coefficients are nonzero. A singular kernel matrix allows coefficient changes in the null space without changing the fitted function.

At prediction time, the sum can omit all zero coefficients, requiring only the number of evaluations of the positive-definite kernel corresponding to retained support vectors rather than all $n$. This need not remove the cost of forming or solving with a dense training kernel matrix. Every misclassified point has $m_i\le 0<1$, including a score-zero tie assigned to the opposite class, so $t_i=1$ and ${\hat{\alpha }}_i=Y_i/(2n\lambda )\ne 0$. Thus many classification errors imply many active support vectors and little prediction-time sparsity benefit. Correctly classified points inside the unit margin can also have nonzero coefficients; a point exactly on the margin need not have a nonzero coefficient.

Fix $C>0$ and use the unnormalized sum convention for the penalty. The linear soft-margin support vector machine solvesEliminating the slack variables of a support vector machine gives $\frac{1}{2}\Vert w{\Vert }^2+C{\sum }_i\max \{0,1-y_{i}f(x_i)\}$, where $f(x)=w^{T}x+b$ and the maximum is the hinge loss. The classifier uses the sign of $f$ and its support-vector-machine decision boundary is $f=0$.

For clarity about dual support vectors and margin degeneracy, let ${\alpha }_i$ be the multipliers of the margin constraints. The Karush-Kuhn-Tucker conditions giveThe dual support vectors are those with ${\alpha }_i>0$. A coefficient strictly between $0$ and $C$ puts a point exactly on its margin boundary; a point with positive slack has ${\alpha }_i=C$. A zero coefficient implies zero slack and signed margin at least one. Geometrically one often counts all points with $y_{i}f(x_i)\le 1$ as support vectors. At a degeneracy, equality can occur with ${\alpha }_i=0$, so the two descriptions need not coincide. Either convention supports the bound in the following part when stated consistently.

For $X_i^{\ast }=(1,X_i^T{)}^T$ and $\gamma =({\gamma }_0,{\beta }^T{)}^T$, a linear support vector machine predictsOne penalized formulation minimizes empirical hinge loss plus a squared Euclidean norm penalty:Conventions often leave the intercept unpenalized, replacing $\Vert \gamma {\Vert }_2^2$ by $\Vert \beta {\Vert }_2^2$; this does not change the role of the two terms.

The hinge loss $\varphi (t)=\max (0,1-t)$ is one-Lipschitz. Centering it at zero and applying the contraction lemma givesFor an empirical risk minimizer and a population minimizer, the standard expected ERM inequality with the $1/n$ Rademacher convention used in part (a) isPart (c) consequently yieldsThus $K=2C^{2}s$ for this normalization of Rademacher complexity.