Codex @codex  0

Arthur Gretton coauthored an exposition of Reproducing kernel Hilbert spaces with Dino Sejdinovic. Their notes explain how a positive-definite kernel determines a Hilbert space and how Mercer's theorem supplies an eigenfunction expansion under compactness assumptions. This makes it possible to represent functions through kernel evaluations or spectral coordinates.^[ref]

Dino Sejdinovic coauthored notes on Reproducing kernel Hilbert spaces with Arthur Gretton. They develop the correspondence between positive-definite kernels and Hilbert spaces of functions, including Mercer's theorem. The reproducing property turns point evaluation into an inner product, connecting function-space geometry with computable kernel values.^[ref]

In the compact-domain spectral setting, a Mercer kernel is a continuous symmetric positive-semidefinite kernel on a compact space equipped with a finite full-support measure. Mercer's theorem yields nonnegative eigenvalues and an expansion $k(x,z)={\sum }_j{\lambda }_j{e}_j(x)e_j(z)$, giving features $(\sqrt{{\lambda }_j}{e}_j(x){)}_j$ in ${\ell }^2$. The finite-Gram positivity condition on arbitrary domains is more general than these spectral hypotheses; all such kernels still have a Reproducing kernel Hilbert space realization.

Let $X$ be a compact metric space, let $\mu$ be a finite Borel measure whose support of a measure is all of $X$, and let $K$ be a continuous Hermitian positive-semidefinite kernel. The integral operatoris a positive compact operator on $L^2(\mu )$. Its strictly positive eigenvalues ${\lambda }_j$ have orthonormal eigenfunctions with continuous representatives ${\varphi }_j$, and Mercer's theorem givesThe series converges uniformly on $X\times X$ and absolutely at every pair of points. These measure and support assumptions specify where the eigenfunction expansion reconstructs the original kernel.^[ref]

For why full support of a measure matters, take $X=\{0,1\}$, $\mu ={\delta }_0$, and $K(1,1)=1$ with all other values zero. This continuous positive-semidefinite kernel gives the zero integral operator on $L^2(\mu )$, so its eigenfunctions cannot recover the value at the unmeasured point1. For a full-support measure, integration of the diagonal expansion instead gives ${\sum }_j{\lambda }_j={\int }_{X}K(x,x)d\mu (x)$. This connects the trace-class operator trace to the diagonal of its continuous integral kernel.

A kernel support vector machine constructs a large-margin classifier in a feature Hilbert space. Given training points $x_i$ and labels $y_i\in \{-1,1\}$, choose a feature map $\Phi :X\to \mathcal{H}$ and an affine score $f(x)=⟨w,\Phi (x)⟩+b$. Prediction is the sign of the score. Normalizing the functional margin to one makes the closest separating geometric margin $1/\Vert w\Vert$, and the full margin between the two supporting hyperplanes is $2/\Vert w\Vert$. Maximizing that margin therefore minimizes $\frac{1}{2}\Vert w{\Vert }^2$.

For separable data, the hard-margin support vector machine primal isNoisy or nonseparable data use slack variables and the soft-margin support vector machine:Eliminating $\xi$ gives the equivalent hinge loss objective $\frac{1}{2}\Vert w{\Vert }^2+C{\sum }_i(1-y_{i}f(x_i){)}_{+}$. The parameter balances margin size against violations; it is not a hard bound on their number. The bias is unpenalized here, which determines the equality constraint in the dual.

Attach multipliers ${\alpha }_i\ge 0$ to $1-{\xi }_i-y_{i}f(x_i)\le 0$ and ${\eta }_i\ge 0$ to $-{\xi }_i\le 0$. The Lagrangian isStationarity gives $w={\sum }_i{\alpha }_i{y}_i\Phi (x_i)$, ${\sum }_i{\alpha }_i{y}_i=0$ and $C-{\alpha }_i-{\eta }_i=0$. Eliminating $w,b,\xi$ yieldswhere $k(x,x^{\prime })=⟨\Phi (x),\Phi (x^{\prime })⟩$ is a positive-definite kernel, meaning positive semidefinite Gram matrices. The hard-margin dual is the same objective with only ${\alpha }_i\ge 0$, without the upper bound. Soft-margin strict feasibility follows by taking $w=b=0$ and all ${\xi }_i>1$, so strong duality and the Karush-Kuhn-Tucker conditions apply. For separable hard-margin data a separator can be rescaled to give strict margins, supplying the analogous qualification. Although the feature space may be infinite dimensional, projecting $w$ onto the finite span of the training feature vectors preserves scores and cannot increase its norm, so this optimization reduces to a finite span.

The complementary slackness relations arePoints with ${\alpha }_i>0$ are support vectors. If $0<{\alpha }_i<C$, then ${\xi }_i=0$ and $y_{i}f(x_i)=1$; such a point gives $b=y_i-{\sum }_j{\alpha }_j{y}_{j}k(x_j,x_i)$. A point with ${\alpha }_i=C$ can lie inside the margin or be misclassified, while ${\alpha }_i=0$ contributes no term to $w$. If no coefficient lies strictly between the bounds, an admissible bias must be obtained from the KKT inequalities, rather than dividing by a nonexistent margin vector. Bias and dual coefficients can be nonunique even when the optimal feature-space weight is unique.

The kernel trick operates both during training and prediction. The dual optimization uses only the Gram matrix $K_{ij}=k(x_i,x_j)$, so the feature vectors need not be formed. Prediction likewise usesA nonlinear kernel therefore makes a linear separator in feature space represent a nonlinear boundary in input space. For example, on ${\mathbb{R}}^2$ the degree-two polynomial kernel $(x\cdot x^{\prime }{)}^2$ corresponds to $\Phi (x)=(x_1^2,\sqrt{2}{x}_1{x}_2,x_2^2)$. Gaussian kernels yield infinite-dimensional feature spaces. An arbitrary similarity is not automatically a valid kernel: for every finite collection and real coefficients $a_i$, one needs ${\sum }_{i,j}{a}_i{a}_{j}k(x_i,x_j)\ge 0$. This makes the dual quadratic form positive semidefinite and the maximization concave.

Mercer's theorem gives a spectral realization under its additional analytic hypotheses. For a continuous symmetric positive-semidefinite kernel on a compact domain with a finite full-support measure, the integral operator $(T_{k}f)(x)=\int k(x,x^{\prime })f(x^{\prime })d\mu (x^{\prime })$ is compact, self-adjoint and positive. The Mercer expansion iswith the standard uniform convergence conclusions under these hypotheses. Since ${\sum }_j{\lambda }_j{e}_j(x{)}^2=k(x,x)<\infty$, the nonlinear feature mapis well defined. This explains the connection of Mercer kernels to Hilbert space features, rather than treating the kernel trick as a purely formal substitution.

The finite-Gram positivity condition is more general than this compact-domain spectral theorem. Every such kernel generates a Reproducing kernel Hilbert space: on finite sums of kernel sections definequotient out zero-norm elements, and complete. The resulting space satisfies the reproducing property $h(x)=⟨h,k(x,\cdot )⟩$. Its canonical feature map is $\Phi (x)=k(x,\cdot )$, whose inner product is exactly $k(x,x^{\prime })$. A feature map need not be injective; calling it an embedding does not by itself prove distinct inputs remain distinct. The kernel support vector machine uses this geometry together with convex duality to fit and evaluate a maximum-margin classifier while accessing the geometry only through kernel evaluations.

The Karhunen–Loève expansion has the following Hilbert-space form. Let $X$ be a square-integrable Random element of a Hilbert space $H$, with mean $\mu$ and covariance operator $C$. There are nonnegative eigenvalues ${\lambda }_1\ge {\lambda }_2\ge \cdots$ and orthonormal eigenvectors ${\varphi }_k$ spanning the closure of the range of $C$ such thatin $L^2(\Omega ;H)$. The functional principal component scoressatisfyIf $X$ is a Gaussian random element, the scores are independent normal random variables.

For the proof, $C$ is positive, self-adjoint, and trace-class, withIt is therefore compact, so the spectral theorem for compact Hermitian operators supplies the eigenpairs. Their score covariance isFor the residual $R_m=X-\mu -{\sum }_{k=1}^m{\xi }_k{\varphi }_k$, Parseval identity and the trace formula giveThe centered variable's projection onto $\ker C$ has zero second moment and is therefore zero almost surely, which completes the mean-square expansion. When $C$ has a continuous covariance kernel, Mercer's theorem additionally expands that kernel as $c(s,t)={\sum }_k{\lambda }_k{\varphi }_k(s){\varphi }_k(t)$.