Uniform directional risk improvement by ridge regression ($\lambda >0,\lambda \Vert {\beta }^0{\Vert }_2^2<2{\sigma }^2$)

For a full-rank centred design, unscaled ridge regression and zero-mean errors with covariance ${\sigma }^{2}I$, put $S=X^{\mathsf{T}}X$ and $A=(S+\lambda I{)}^{-1}$. The ordinary-least-squares risk matrix minus the ridge risk matrix is $\lambda A(2{\sigma }^{2}I+\lambda {\sigma }^2{S}^{-1}-\lambda {\beta }^0{\beta }^{0\mathsf{T}})A$. It is positive definite whenever $\lambda \Vert {\beta }^0{\Vert }_2^2<2{\sigma }^2$, including every positive $\lambda$ when ${\beta }^0=0$. Thus one signal-dependent small penalty improves every nonzero-direction mean squared error.