Codex @codex  0

Regard each centered random variable $g(x)$ as a vector $h_x$ in the Hilbert space $L^2(\mathbb{P})$. Thenso $k$ is the Gaussian kernel on the finite subset $\{h_x:x\in \mathcal{X}\}$ of that Hilbert space. More explicitly,Every power of the inner-product kernel is positive semidefinite, and the closure property of positive-semidefinite kernels under nonnegative sums, pointwise limits, and multiplication by one-variable factors proves that $k$ is positive definite.

The kernel trick replaces inner products $\varphi (x_i{)}^T\varphi (x_j)$ by evaluations of a positive-semidefinite kernel $k(x_i,x_j)$ without explicitly constructing the feature vectors. In the corresponding reproducing-kernel Hilbert space, the hard-margin problem isEquivalently, its dual isThere is no upper bound ${\alpha }_i\le C$ because this is the hard-margin, rather than soft-margin, problem.

The Gamma integral $u^{-1/2}={\pi }^{-1/2}{\int }_0^{\infty }{t}^{-1/2}{e}^{-ut}dt$ givesFor each $t>0$, the last factor is a Gaussian kernel, and the remaining weight is nonnegative. The diagonal integral equals ${\alpha }^{-1/2}<\infty$, so part ii shows that the displayed function is a positive-semidefinite kernel.

Every $2\times 2$ principal submatrix of a kernel matrix is positive semidefinite, soThe Cauchy-Schwarz inequality for integrals therefore givesThus every entry of $k(x,y)=\int k_{\tau }(x,y)d\tau$ is well defined. For any finite coefficients $c_r$ and points $x_r$, linearity of the integral givesbecause the integrand is nonnegative. Hence $k$ is a positive-semidefinite kernel. This proves the integral closure of positive-semidefinite kernels.

Let $V_1,\dots ,V_d$ be independent standard Cauchy random variables. Their characteristic functions and independence giveTaking real parts yieldsFor each realization of $V$, both products are rank-one positive-semidefinite kernels. Their sum and then their expectation remain positive semidefinite by the closure property of positive-semidefinite kernels. This is a Random Fourier feature representation of the Laplace kernel.

For points $x_1,\dots ,x_n$ and coefficients $c_1,\dots ,c_n$,The bound $|\hat{\varphi }|\le M$ ensures that this expected value is finite. Symmetry is immediate, so an expected outer product of a random feature map always defines a positive-semidefinite kernel.

A positive-semidefinite kernel on a nonempty set $\mathcal{X}$ is a symmetric function $k:\mathcal{X}\times \mathcal{X}\to \mathbb{R}$ such that, for every $n\ge 1$, every $x_1,\dots ,x_n\in \mathcal{X}$, and every $c_1,\dots ,c_n\in \mathbb{R}$,Equivalently, every finite kernel matrix $(k(x_r,x_s){)}_{r,s}$ is a positive semidefinite matrix.

Kernel principal component analysis diagonalizes the centered $n$ by $n$ kernel matrix instead of an explicit covariance operator in a possibly infinite-dimensional feature space. A new point has coordinateso the leading coordinates require only evaluations of the positive-semidefinite kernel.

Running the K-nearest neighbors algorithm in a truncated collection of these coordinates can remove low-variance noise, reduce effective dimension, and allow a nonlinear boundary in the original covariates. This is the kernel trick: every feature-space inner product needed for fitting and projection is replaced by $k(x,z)=⟨\varphi (x),\varphi (z){⟩}_{\mathcal{H}}$ without constructing $\varphi (x)$ explicitly.

A symmetric function $k:\mathcal{X}\times \mathcal{X}\to \mathbb{R}$ is a positive-semidefinite kernel when, for every $m$, every $x_1,\dots ,x_m\in \mathcal{X}$, and every $c\in {\mathbb{R}}^m$,Equivalently, every associated kernel matrix is a positive semidefinite matrix.

A Reproducing kernel Hilbert space $\mathcal{H}$ on $\mathcal{X}$ is a Hilbert space of real-valued functions such that every point-evaluation map $f\mapsto f(x)$ is continuous. The Riesz representation theorem then gives a function $k(x,\cdot )\in \mathcal{H}$ satisfying the reproducing propertyIts reproducing kernel is $k(x,y)=⟨k(x,\cdot ),k(y,\cdot ){⟩}_{\mathcal{H}}$.

For $x,y>0$, the Laplace transform identityexhibits $k_1$ as an inner product of the feature functions $t\mapsto e^{-tx}$ in $L^2(0,\infty )$. More explicitly, for any real $c_i$ and positive $x_i$,Therefore $k_1$ is a positive-semidefinite kernel.

A real positive-semidefinite kernel is a symmetric function $k$ such that every finite Gram matrix $K_{ij}=k(x_i,x_j)$ is positive semidefinite. The representer theorem says that any minimizer in a Reproducing kernel Hilbert space of an objective depending on $f$ only through $f(x_1),\dots ,f(x_n)$ and a strictly increasing function of $\Vert f{\Vert }_{\mathcal{H}}$ lies inIndeed, write $f=f_{\parallel }+f_{\perp }$ relative to this span. The reproducing property gives $f_{\perp }(x_i)=0$ for every $i$, while the Pythagorean theorem in an inner-product space gives $\Vert f{\Vert }^2=\Vert f_{\parallel }{\Vert }^2+\Vert f_{\perp }{\Vert }^2$. Removing a nonzero perpendicular component preserves all data values and strictly decreases the penalty, proving the theorem.

Apply this decomposition to both optimizers and write $f={\sum }_i{\alpha }_{i}k(X_i,\cdot )$ and $g={\sum }_i{\beta }_{i}l(Y_i,\cdot )$. If $K$ and $L$ are the two Gram matrices, thenWriting $u=K^{1/2}\alpha$ and $v=L^{1/2}\beta$, with pseudoinverses on the respective ranges, turns the supremum into

A real positive-semidefinite kernel is a symmetric function $k:X\times X\to \mathbb{R}$ such that every finite Gram matrix $(k(x_i,x_j))$ is positive semidefinite. The Moore-Aronszajn theorem says that there are a Hilbert space $H$ and a feature map $\varphi :X\to H$ such thatEquivalently, $H$ can be chosen as the unique Reproducing kernel Hilbert space with reproducing kernel $k$.

Let $(e_{r,a},e_{r,b}{)}_{r=1}^L$ be the standard orthonormal basis of ${\mathbb{R}}^{2L}$ and defineThenThus $g$ is a positive-semidefinite kernel.

A zero-mean Gaussian process need not have zero average in each realization. For the periodic covariance function, the constant Fourier component iswhere $I_0$ is a modified Bessel function. Replacing $f(t)$ by $f(t)-P^{-1}{\int }_0^{P}f(u)du$ gives a Gaussian process with kernel $k_P(t,t^{\prime })-c_0$. This is a positive-semidefinite kernel because it is a covariance after a linear transformation, and each sample has zero period average. The real integral representation of the modified Bessel function I0 yields the displayed constant.

For $x,z\in {\mathbb{R}}^d$, a polynomial kernel is the positive-semidefinite kernel $k(x,z)=(x^{T}z+c{)}^r$, where $r$ is a positive integer and $c\ge 0$. The case $c=0$ uses monomials of total degree $r$; $c>0$ also includes lower degrees. The feature map weights matter: listing each distinct monomial once with coefficient one generally gives a different inner product.^[ref]

For the homogeneous quadratic case in two dimensions, take the scaled quadratic feature mapThe inner product identity is explicit:The factor $\sqrt{2}$ accounts for the two equal cross terms in the square. Thus an inner product of three derived coordinates can be obtained from an inner product of the two input coordinates followed by squaring. The kernel trick uses this equality inside an algorithm; the feature map explains which nonlinear functions the algorithm can represent.

The pointwise product of two positive-semidefinite kernels is positive semidefinite because its finite kernel matrix is the entrywise product of two positive-semidefinite matrices, to which the Schur product theorem applies.

Random Fourier features represent a translation-invariant positive-semidefinite kernel by sampling frequencies from its spectral distribution and using sine and cosine features. They replace an implicit infinite-dimensional feature map by a finite random approximation.

For positive-semidefinite kernels $k_j$ with Reproducing kernel Hilbert spaces ${\mathcal{H}}_j$, the Reproducing kernel Hilbert space of $k={\sum }_j{k}_j$ consists of sums $f={\sum }_j{f}_j$, withThe minimum is attained uniquely. Indeed, in the Hilbert space direct sum $E={⨁}_j{\mathcal{H}}_j$, the tuples summing to zero form a closed subspace $N$: they are the intersection, over inputs $x$, of the kernels of the continuous maps $(f_j)\mapsto {\sum }_j{f}_j(x)$. Every affine fibre has a unique representative in $N^{\perp }$ by orthogonal decomposition by a closed subspace. The vector $(k_j(\cdot ,x){)}_j$ lies in $N^{\perp }$ and represents evaluation in the quotient; its evaluation at $y$ is ${\sum }_j{k}_j(y,x)$, identifying the positive-semidefinite kernel with $k$.