= Prediction-risk bound for kernel ridge regression
Fix observation points and suppose $Y_i=f_0(x_i)+\varepsilon_i$, where the noise has zero <expected value> and <covariance matrix> $\sigma^2I$. Use the unscaled objective $\sum_i(Y_i-f(x_i))^2+\lambda\|f\|_{\mathcal H}^2$ with $\lambda>0$ and $f_0\in\mathcal H$. The <kernel-ridge hat matrix> is now $H=K(K+\lambda I)^{-1}$. For $g_i=f_0(x_i)$,
$$
\frac1n\mathbb E\|HY-g\|^2=\frac1n\|(I-H)g\|^2+\frac{\sigma^2}n\operatorname{tr}(H^2).
$$
This is prediction error at the fixed observation points, with separate squared-bias and <variance> terms.
For <eigenvalues> $d_i\geq0$ of the <kernel matrix>,
$$
\frac{d_i^2}{(d_i+\lambda)^2}\leq\min\!\left\{\frac{d_i}{4\lambda},1\right\},\qquad
\frac{\lambda^2d_i}{(d_i+\lambda)^2}\leq\frac\lambda4.
$$
The <reproducing property> and <Bessel inequality> represent $g=K^{1/2}v$ with $\|v\|^2\leq\|f_0\|_{\mathcal H}^2$, including singular $K$. Hence
$$
\frac1n\mathbb E\|HY-g\|^2\leq\frac{\lambda}{4n}\|f_0\|_{\mathcal H}^2+\frac{\sigma^2}n\sum_i\min\!\left\{\frac{d_i}{4\lambda},1\right\}.
$$
The bound makes the opposing effects of the ridge explicit: it increases the displayed bias bound while reducing the <variance> bound.
Back to article page