 Kernel trick

The kernel trick evaluates inner products of feature map outputs through $k(x,z)=⟨\varphi (x),\varphi (z)⟩$ without explicitly constructing those vectors. An algorithm expressed entirely through these inner products can therefore operate on nonlinear derived coordinates using only evaluations of a positive-semidefinite kernel on the original inputs. A classifier linear in the derived coordinates can have a nonlinear boundary in the original coordinates.

The polynomial kernel gives a finite, fully visible example: $k(x,z)=(x^{T}z{)}^2$ corresponds to $\varphi (x)=(x_1^2,\sqrt{2}{x}_1{x}_2,x_2^2{)}^T$. The XOR separation by a product feature below makes the interaction coordinate $x_1{x}_2$ useful, while dual prediction with a kernel shows how to predict without calculating it explicitly.

An arbitrary score called a similarity need not be a positive-semidefinite kernel. Symmetry alone, even with nonnegative entries, is insufficient: the symmetric matrixhas $a^{T}Sa=-2$ for $a=(1,-1{)}^T$, so it cannot be a Gram matrix of real inner products. A positive-semidefinite kernel requires a positive semidefinite matrix for every finite choice of inputs, not just the observed training data.^[ref] The existing Moore-Aronszajn theorem connects this requirement to a Reproducing kernel Hilbert space; this example needs only finite linear algebra.