= Uniform directional risk improvement by ridge regression
{title2=$\lambda>0,\quad\lambda\|\beta^0\|_2^2<2\sigma^2$}
For a full-rank centred design, unscaled <ridge regression> and zero-mean errors with covariance $\sigma^2I$, put $S=X^{\mathsf T}X$ and $A=(S+\lambda I)^{-1}$. The ordinary-least-squares risk matrix minus the ridge risk matrix is $\lambda A(2\sigma^2I+\lambda\sigma^2S^{-1}-\lambda\beta^0\beta^{0\mathsf T})A$. It is positive definite whenever $\lambda\|\beta^0\|_2^2<2\sigma^2$, including every positive $\lambda$ when $\beta^0=0$. Thus one signal-dependent small penalty improves every nonzero-direction <mean squared error>.
Back to article page
