Past exam of the mathematics course of the University of Cambridge / 2021 / iii / Paper 205 / 4

Regard each centered random variable $g(x)$ as a vector $h_x$ in the Hilbert space $L^2(\mathbb{P})$. Thenso $k$ is the Gaussian kernel on the finite subset $\{h_x:x\in \mathcal{X}\}$ of that Hilbert space. More explicitly,Every power of the inner-product kernel is positive semidefinite, and the closure property of positive-semidefinite kernels under nonnegative sums, pointwise limits, and multiplication by one-variable factors proves that $k$ is positive definite.

The kernel ridge regression estimator minimizesBy the representer theorem, its fitted-value vector isWriting $f$ for the true value vector, the variance contribution to $\mathbb{E}\Vert f-\hat{f}{\Vert }_2^2$ isThe squared bias is $\Vert \lambda (K+\lambda I{)}^{-1}f{\Vert }_2^2$. In an orthonormal eigenbasis of $K$, the scalar inequalitywhich is equivalent to $4d\lambda \le (d+\lambda {)}^2$, yieldsFor $\Vert f{\Vert }_2=1$, only the last term varies. The Rayleigh quotient of $K^{-1}$ is maximized by a unit eigenvector associated with its largest eigenvalue $1/d_n$, equivalently an eigenvector of $K$ associated with $d_n$.