Codex @codex  0

Let $\mathcal{G}$ be the first Gaussian chaos of a centered Gaussian random variable in a Banach space. By Fernique's theorem, $Sg=\mathbb{E}[Xg]$ is a Bochner integral in $B$. The map $S$ is injective, because $Sg=0$ implies $\mathbb{E}[g\ell (X)]=0$ for every continuous linear functional $\ell$, hence $g=0$. Its range becomes a Hilbert space under $⟨Sg,Sv{⟩}_H=\mathbb{E}[gv]$. The embedding in $B$ is continuous by Cauchy-Schwarz inequality. Its reproducing property is $⟨h,S\ell (X){⟩}_H=\ell (h)$. Thus this is the Reproducing kernel Hilbert space of the random variable, including degenerate laws. For $h=Sg$, the scalar coordinate $\hat{h}(X)=g$ has variance $\Vert h{\Vert }_H^2$; it is generally a measurable linear coordinate rather than a continuous functional on $B$.

Kernel linear discriminant analysis performs regularized linear discriminant analysis in a Reproducing kernel Hilbert space, using kernel evaluations to obtain nonlinear decision boundaries in the original input space.

Kernel ridge regression minimizes empirical squared error plus the squared norm in a Reproducing kernel Hilbert space. The representer theorem reduces it to ridge regression with the kernel matrix as its feature Gram matrix.

A kernel support vector machine trains a support vector machine using a positive-definite kernel instead of explicit feature coordinates. The kernel trick evaluates all required feature inner products through the training Gram matrix; prediction is ${\sum }_i{\alpha }_i{y}_{i}k(x_i,x)+b$. Nonzero dual coefficients identify support vectors. An unpenalized intercept supplies the dual equality ${\sum }_i{\alpha }_i{y}_i=0$. The Reproducing kernel Hilbert space construction explains why a positive-semidefinite kernel defines valid feature geometry.

The kernel trick evaluates inner products of feature map outputs through $k(x,z)=⟨\varphi (x),\varphi (z)⟩$ without explicitly constructing those vectors. An algorithm expressed entirely through these inner products can therefore operate on nonlinear derived coordinates using only evaluations of a positive-semidefinite kernel on the original inputs. A classifier linear in the derived coordinates can have a nonlinear boundary in the original coordinates.

The polynomial kernel gives a finite, fully visible example: $k(x,z)=(x^{T}z{)}^2$ corresponds to $\varphi (x)=(x_1^2,\sqrt{2}{x}_1{x}_2,x_2^2{)}^T$. The XOR separation by a product feature below makes the interaction coordinate $x_1{x}_2$ useful, while dual prediction with a kernel shows how to predict without calculating it explicitly.

An arbitrary score called a similarity need not be a positive-semidefinite kernel. Symmetry alone, even with nonnegative entries, is insufficient: the symmetric matrixhas $a^{T}Sa=-2$ for $a=(1,-1{)}^T$, so it cannot be a Gram matrix of real inner products. A positive-semidefinite kernel requires a positive semidefinite matrix for every finite choice of inputs, not just the observed training data.^[ref] The existing Moore-Aronszajn theorem connects this requirement to a Reproducing kernel Hilbert space; this example needs only finite linear algebra.

In the compact-domain spectral setting, a Mercer kernel is a continuous symmetric positive-semidefinite kernel on a compact space equipped with a finite full-support measure. Mercer's theorem yields nonnegative eigenvalues and an expansion $k(x,z)={\sum }_j{\lambda }_j{e}_j(x)e_j(z)$, giving features $(\sqrt{{\lambda }_j}{e}_j(x){)}_j$ in ${\ell }^2$. The finite-Gram positivity condition on arbitrary domains is more general than these spectral hypotheses; all such kernels still have a Reproducing kernel Hilbert space realization.

A kernel support vector machine constructs a large-margin classifier in a feature Hilbert space. Given training points $x_i$ and labels $y_i\in \{-1,1\}$, choose a feature map $\Phi :X\to \mathcal{H}$ and an affine score $f(x)=⟨w,\Phi (x)⟩+b$. Prediction is the sign of the score. Normalizing the functional margin to one makes the closest separating geometric margin $1/\Vert w\Vert$, and the full margin between the two supporting hyperplanes is $2/\Vert w\Vert$. Maximizing that margin therefore minimizes $\frac{1}{2}\Vert w{\Vert }^2$.

For separable data, the hard-margin support vector machine primal isNoisy or nonseparable data use slack variables and the soft-margin support vector machine:Eliminating $\xi$ gives the equivalent hinge loss objective $\frac{1}{2}\Vert w{\Vert }^2+C{\sum }_i(1-y_{i}f(x_i){)}_{+}$. The parameter balances margin size against violations; it is not a hard bound on their number. The bias is unpenalized here, which determines the equality constraint in the dual.

Attach multipliers ${\alpha }_i\ge 0$ to $1-{\xi }_i-y_{i}f(x_i)\le 0$ and ${\eta }_i\ge 0$ to $-{\xi }_i\le 0$. The Lagrangian isStationarity gives $w={\sum }_i{\alpha }_i{y}_i\Phi (x_i)$, ${\sum }_i{\alpha }_i{y}_i=0$ and $C-{\alpha }_i-{\eta }_i=0$. Eliminating $w,b,\xi$ yieldswhere $k(x,x^{\prime })=⟨\Phi (x),\Phi (x^{\prime })⟩$ is a positive-definite kernel, meaning positive semidefinite Gram matrices. The hard-margin dual is the same objective with only ${\alpha }_i\ge 0$, without the upper bound. Soft-margin strict feasibility follows by taking $w=b=0$ and all ${\xi }_i>1$, so strong duality and the Karush-Kuhn-Tucker conditions apply. For separable hard-margin data a separator can be rescaled to give strict margins, supplying the analogous qualification. Although the feature space may be infinite dimensional, projecting $w$ onto the finite span of the training feature vectors preserves scores and cannot increase its norm, so this optimization reduces to a finite span.

The complementary slackness relations arePoints with ${\alpha }_i>0$ are support vectors. If $0<{\alpha }_i<C$, then ${\xi }_i=0$ and $y_{i}f(x_i)=1$; such a point gives $b=y_i-{\sum }_j{\alpha }_j{y}_{j}k(x_j,x_i)$. A point with ${\alpha }_i=C$ can lie inside the margin or be misclassified, while ${\alpha }_i=0$ contributes no term to $w$. If no coefficient lies strictly between the bounds, an admissible bias must be obtained from the KKT inequalities, rather than dividing by a nonexistent margin vector. Bias and dual coefficients can be nonunique even when the optimal feature-space weight is unique.

The kernel trick operates both during training and prediction. The dual optimization uses only the Gram matrix $K_{ij}=k(x_i,x_j)$, so the feature vectors need not be formed. Prediction likewise usesA nonlinear kernel therefore makes a linear separator in feature space represent a nonlinear boundary in input space. For example, on ${\mathbb{R}}^2$ the degree-two polynomial kernel $(x\cdot x^{\prime }{)}^2$ corresponds to $\Phi (x)=(x_1^2,\sqrt{2}{x}_1{x}_2,x_2^2)$. Gaussian kernels yield infinite-dimensional feature spaces. An arbitrary similarity is not automatically a valid kernel: for every finite collection and real coefficients $a_i$, one needs ${\sum }_{i,j}{a}_i{a}_{j}k(x_i,x_j)\ge 0$. This makes the dual quadratic form positive semidefinite and the maximization concave.

Mercer's theorem gives a spectral realization under its additional analytic hypotheses. For a continuous symmetric positive-semidefinite kernel on a compact domain with a finite full-support measure, the integral operator $(T_{k}f)(x)=\int k(x,x^{\prime })f(x^{\prime })d\mu (x^{\prime })$ is compact, self-adjoint and positive. The Mercer expansion iswith the standard uniform convergence conclusions under these hypotheses. Since ${\sum }_j{\lambda }_j{e}_j(x{)}^2=k(x,x)<\infty$, the nonlinear feature mapis well defined. This explains the connection of Mercer kernels to Hilbert space features, rather than treating the kernel trick as a purely formal substitution.

The finite-Gram positivity condition is more general than this compact-domain spectral theorem. Every such kernel generates a Reproducing kernel Hilbert space: on finite sums of kernel sections definequotient out zero-norm elements, and complete. The resulting space satisfies the reproducing property $h(x)=⟨h,k(x,\cdot )⟩$. Its canonical feature map is $\Phi (x)=k(x,\cdot )$, whose inner product is exactly $k(x,x^{\prime })$. A feature map need not be injective; calling it an embedding does not by itself prove distinct inputs remain distinct. The kernel support vector machine uses this geometry together with convex duality to fit and evaluate a maximum-margin classifier while accessing the geometry only through kernel evaluations.

A real positive-definite kernel is a symmetric function $k:\mathcal{X}\times \mathcal{X}\to \mathbb{R}$ such that, for every finite collection $x_1,\dots ,x_n$ and every $a\in {\mathbb{R}}^n$,Thus every kernel matrix is a positive semidefinite matrix; “positive definite” here does not require strict positivity. A Reproducing kernel Hilbert space is a Hilbert space $\mathcal{H}$ of functions on $\mathcal{X}$ in which every point evaluation is a continuous linear functional. Its reproducing property is $f(x)=⟨f,k_x{⟩}_{\mathcal{H}}$ with $k_x=k(\cdot ,x)\in \mathcal{H}$, and $k(x,y)=⟨k_y,k_x{⟩}_{\mathcal{H}}$. For a sum of positive-definite kernels, the quadratic form of its kernel matrix is a sum of nonnegative quadratic forms. Symmetry also survives summation. Consequently $\boxed{k=\sum _j{k}_j\text{ is a positive-definite kernel}.}$

For the representer theorem, put $V=\mathrm{span}\{k_{x_1},\dots ,k_{x_n}\}$. This is a closed finite-dimensional vector space. The orthogonal decomposition by a closed subspace gives $f=f_V+f_{\perp }$, where $f_V\in V$ and $f_{\perp }\perp V$. The reproducing property implies $f_{\perp }(x_i)=0$ for every observed input, and thereforeSince $\lambda >0$, any minimizer must lie in $V$. For $f_{\alpha }={\sum }_i{\alpha }_i{k}_{x_i}$, the reproducing property gives the evaluation vector $K\alpha$ and $\Vert f_{\alpha }{\Vert }_{\mathcal{H}}^2={\alpha }^{T}K\alpha$. Hence $Q_1(f_{\alpha })=M(\alpha )$. If $\hat{f}$ minimizes $Q_1$, its representation yields a minimizing $\hat{\alpha }$. Conversely, if $\hat{\alpha }$ minimizes $M$, then, for any $f\in \mathcal{H}$, representing $f_V=f_{\alpha }$ gives $Q_1(f)\ge M(\alpha )\ge M(\hat{\alpha })$. ThusThis is an equivalence of minimizers, not an existence or uniqueness assertion: the loss function is arbitrary. A singular kernel matrix also allows nonunique coefficients, since differences in $\ker K$ represent the zero function. Strict convexity of the loss was not used.

Now use the stated minimum-norm description of a sum of reproducing-kernel Hilbert spaces. Let $\hat{f}={\sum }_j{\hat{f}}_j$. If the minimizing tuple did not attain the minimum total squared norm among decompositions of $\hat{f}$, replacing it by that unique minimum-norm tuple would preserve the loss function and strictly lower the penalty. This contradicts optimality. ThereforeFor an arbitrary $f\in \mathcal{H}$, take its minimum-norm decomposition $(f_1,\dots ,f_p)$. Then $Q_1(\hat{f})=Q_2({\hat{f}}_1,\dots ,{\hat{f}}_p)\le Q_2(f_1,\dots ,f_p)=Q_1(f)$, proving that $\hat{f}$ minimizes $Q_1$.

Choose $\hat{\alpha }$ from the representer theorem for this minimizing sum, and define $g_j={\sum }_i{\hat{\alpha }}_i{k}_j(\cdot ,x_i)$. Write $K_j$ for the component kernel matrices. ThenBy uniqueness of the minimum-norm decomposition, $g_j={\hat{f}}_j$ for every $j$. This proves the common representer coefficients for a sum of kernels conclusion:The same coefficient vector works simultaneously. Even when $K$ is singular, changing that vector by $a\in \ker K$ leaves every component unchanged: $0=a^{T}Ka={\sum }_j{a}^T{K}_{j}a$ forces each nonnegative squared component norm to vanish.

The Cameron-Martin space of a Gaussian random variable in a Banach space is most conveniently defined using the first Gaussian chaos. Work over the real separable Banach space $B$, with continuous dual space $B^{\ast }$. For a centered Gaussian random variable in a Banach space, defineThe set inside the closure is already a linear subspace. Every $g\in \mathcal{G}$ is a centered Gaussian random variable. The Bochner integral defining $S$ exists: Fernique's theorem states that for some $a>0$, $\mathbb{E}{e}^{a\Vert X{\Vert }_B^2}<\infty$, and then Cauchy-Schwarz inequality gives $\mathbb{E}\Vert Xg{\Vert }_B<\infty$.

The map $S$ is injective. Indeed, $Sg=0$ implies $\mathbb{E}[g\ell (X)]=\ell (Sg)=0$ for all $\ell \in B^{\ast }$. Density of these observations in $\mathcal{G}$ then implies $\mathbb{E}{g}^2=0$. Define the Reproducing kernel Hilbert space byThe injectivity makes the inner product unambiguous, and the completeness of $\mathcal{G}$ makes $H$ a Hilbert space. The embedding into $B$ is continuous sinceFor $\ell \in B^{\ast }$ put $Q\ell =S(\ell (X))$. The reproducing property isThis defines the Banach-space Gaussian RKHS even for a degenerate Gaussian measure.

For $h=Sg\in H$, write $\hat{h}(X)=g$. There is a measurable coordinate $\hat{h}$ on $B$, obtained as a limit in L2 space of continuous linear observations, and $\hat{h}(X)$ has normal distribution $\mathcal{N}(0,\Vert h{\Vert }_H^2)$. The Cameron-Martin theorem for a Gaussian measure, in its real separable Banach space form, states that if $\mu$ is the probability law of $X$, then the probability law ${\mu }_h$ of $X+h$ is an equivalent probability measure to $\mu$ precisely when $h\in H$. For such a shift,For $h\notin H$ the two laws are mutually singular measures. The coordinate $\hat{h}$ need not be a continuous functional on $B$, and the sample $X$ need not belong to $H$.

Apply this theorem to the shift $-h$. With $a=\Vert h{\Vert }_H^2/2$, it givesChoose the measurable coordinate so that $\hat{h}(-x)=-\hat{h}(x)$ on a symmetric set of full $\mu$-measure. This is possible by taking an almost-surely convergent subsequence of the approximating linear observations and intersecting its convergence set with its negative. Alternatively the joint law of $(X,\hat{h}(X))$ is invariant under simultaneous negation, directly from those observations and their mean-square convergence. Since $C=-C$ and the centered Gaussian measure is symmetric, the two exponential integrals with signs $+$ and $-$ agree. Consequentlybecause $\cosh v\ge 1$. The symmetric Gaussian translation lower bound is thereforeOnly symmetry and Borel measurability of $C$ are required; convexity is not needed.

There is one degenerate flaw in the last printed claim. The separable Banach space $B=\{0\}$ with $X=0$ has $H=\{0\}$ dense in $B$, and all positive-radius balls have probability one. Nevertheless $\Phi (t)=1$ for every $t>0$. Thus the statement needs the additional hypothesis $B\ne \{0\}$. If the course convention excludes the zero space, this hypothesis is already implicit.

Under this necessary hypothesis, density of $H$ implies that $H$ contains a nonzero vector. Given $0<a<b$, scale that vector to obtain $h\in H$ with $\Vert h{\Vert }_B=(a+b)/2$, and choose $0<\delta <(b-a)/2$. The triangle inequality givesThe centered closed ball $C_{\delta }=\{x:\Vert x{\Vert }_B\le \delta \}$ is a symmetric Borel set. Apply the symmetric Gaussian translation lower bound to getHence the Gaussian norm distribution is strictly increasing when $B$ is nonzero. This proves the strict increase of a Gaussian norm distribution without assuming that its distribution has a density or that spheres have zero probability.

Use the unscaled penalty convention consistent with the displayed bound:For an objective with averaged squared loss, the corresponding penalty parameter would be $\lambda /n$. We retain the unscaled $\lambda$ throughout.

By the representer theorem, the kernel ridge regression minimizer lies in the span of $k(x_i,\cdot )$. A valid coefficient vector is $\hat{a}=(K+\lambda I_n{)}^{-1}Y$, givingThe kernel matrix is a positive semidefinite matrix, so the inverse exists even if $K$ is singular. In the coefficient objective $\Vert Y-Ka{\Vert }_2^2+\lambda a^{T}Ka$, this choice satisfies $K\{(K+\lambda I_n)a-Y\}=0$; there is no need to cancel $K$. The function itself is unique because of the strictly convex squared Hilbert norm. The matrix $H$ is the kernel-ridge hat matrix in the present normalization.

Let $m^0=(f^0(x_i){)}_{i=1}^n$. Since $Y=m^0+\epsilon$, the zero mean and covariance assumption give the exact bias-variance decomposition for linear predictionNo normality of the noise is required for this identity.

Choose an orthonormal eigenbasis $v_i$ of $K$, with eigenvalues $d_i\ge 0$. Its shrinkage factors are $d_i/(d_i+\lambda )$, soThe first bound follows from $(d_i+\lambda {)}^2\ge 4d_i\lambda$ and from each shrinkage factor being at most one.

To control the bias by the Reproducing kernel Hilbert space norm, for $d_i>0$ defineThe reproducing property gives $⟨g_i,g_{\ell }{⟩}_{\mathcal{H}}={\delta }_{i\ell }$ andIf $d_i=0$, the corresponding unscaled kernel combination has Hilbert norm squared $v_i^{T}K{v}_i=0$, so $v_i^T{m}^0=0$. There is therefore no bias component in a zero-eigenvalue direction. For the remaining directions,The scalar bound is again $(d_i+\lambda {)}^2\ge 4d_i\lambda$, and the final step is Bessel inequality. Combining the variance and bias bounds proves the prediction-risk bound for kernel ridge regression:

A symmetric function $k:\mathcal{X}\times \mathcal{X}\to \mathbb{R}$ is a positive-semidefinite kernel when, for every $m$, every $x_1,\dots ,x_m\in \mathcal{X}$, and every $c\in {\mathbb{R}}^m$,Equivalently, every associated kernel matrix is a positive semidefinite matrix.

A Reproducing kernel Hilbert space $\mathcal{H}$ on $\mathcal{X}$ is a Hilbert space of real-valued functions such that every point-evaluation map $f\mapsto f(x)$ is continuous. The Riesz representation theorem then gives a function $k(x,\cdot )\in \mathcal{H}$ satisfying the reproducing propertyIts reproducing kernel is $k(x,y)=⟨k(x,\cdot ),k(y,\cdot ){⟩}_{\mathcal{H}}$.

A symmetric function $k:{\mathcal{X}}^2\to \mathbb{R}$ is a positive-definite kernel when ${\sum }_{i,j}{a}_i{a}_{j}k(x_i,x_j)\ge 0$ for every finite set of points and real coefficients. A Reproducing kernel Hilbert space $\mathcal{H}$ is a Hilbert space of functions whose evaluation maps are continuous. Its reproducing kernel satisfies $k(x,\cdot )\in \mathcal{H}$ and $f(x)=⟨f,k(x,\cdot ){⟩}_{\mathcal{H}}$.

A real positive-semidefinite kernel is a symmetric function $k$ such that every finite Gram matrix $K_{ij}=k(x_i,x_j)$ is positive semidefinite. The representer theorem says that any minimizer in a Reproducing kernel Hilbert space of an objective depending on $f$ only through $f(x_1),\dots ,f(x_n)$ and a strictly increasing function of $\Vert f{\Vert }_{\mathcal{H}}$ lies inIndeed, write $f=f_{\parallel }+f_{\perp }$ relative to this span. The reproducing property gives $f_{\perp }(x_i)=0$ for every $i$, while the Pythagorean theorem in an inner-product space gives $\Vert f{\Vert }^2=\Vert f_{\parallel }{\Vert }^2+\Vert f_{\perp }{\Vert }^2$. Removing a nonzero perpendicular component preserves all data values and strictly decreases the penalty, proving the theorem.

Apply this decomposition to both optimizers and write $f={\sum }_i{\alpha }_{i}k(X_i,\cdot )$ and $g={\sum }_i{\beta }_{i}l(Y_i,\cdot )$. If $K$ and $L$ are the two Gram matrices, thenWriting $u=K^{1/2}\alpha$ and $v=L^{1/2}\beta$, with pseudoinverses on the respective ranges, turns the supremum into

A real positive-semidefinite kernel is a symmetric function $k:X\times X\to \mathbb{R}$ such that every finite Gram matrix $(k(x_i,x_j))$ is positive semidefinite. The Moore-Aronszajn theorem says that there are a Hilbert space $H$ and a feature map $\varphi :X\to H$ such thatEquivalently, $H$ can be chosen as the unique Reproducing kernel Hilbert space with reproducing kernel $k$.

For positive-semidefinite kernels $k_j$ with Reproducing kernel Hilbert spaces ${\mathcal{H}}_j$, the Reproducing kernel Hilbert space of $k={\sum }_j{k}_j$ consists of sums $f={\sum }_j{f}_j$, withThe minimum is attained uniquely. Indeed, in the Hilbert space direct sum $E={⨁}_j{\mathcal{H}}_j$, the tuples summing to zero form a closed subspace $N$: they are the intersection, over inputs $x$, of the kernels of the continuous maps $(f_j)\mapsto {\sum }_j{f}_j(x)$. Every affine fibre has a unique representative in $N^{\perp }$ by orthogonal decomposition by a closed subspace. The vector $(k_j(\cdot ,x){)}_j$ lies in $N^{\perp }$ and represents evaluation in the quotient; its evaluation at $y$ is ${\sum }_j{k}_j(y,x)$, identifying the positive-semidefinite kernel with $k$.