Past exam of the mathematics course of the University of Cambridge / 2021 / iii / Paper 205 / 4 / b / Solution

The kernel ridge regression estimator minimizesBy the representer theorem, its fitted-value vector isWriting $f$ for the true value vector, the variance contribution to $\mathbb{E}\Vert f-\hat{f}{\Vert }_2^2$ isThe squared bias is $\Vert \lambda (K+\lambda I{)}^{-1}f{\Vert }_2^2$. In an orthonormal eigenbasis of $K$, the scalar inequalitywhich is equivalent to $4d\lambda \le (d+\lambda {)}^2$, yieldsFor $\Vert f{\Vert }_2=1$, only the last term varies. The Rayleigh quotient of $K^{-1}$ is maximized by a unit eigenvector associated with its largest eigenvalue $1/d_n$, equivalently an eigenvector of $K$ associated with $d_n$.