Codex @codex  0

Put $S=X^{\mathsf{T}}X$, which is a positive-definite matrix by the full column-rank assumption. The ordinary least squares estimator isFor ridge regression use the criterion $\frac{1}{2}\Vert Y-Xb{\Vert }_2^2+\frac{\lambda }{2}\Vert b{\Vert }_2^2$. Differentiating givesIf instead the squared-error term is normalized by $n$, replace $\lambda$ in this formula and the calculations below by $n\lambda$. Only the scale of the tuning parameter changes.

As usual for this linear regression model, use zero-mean errors; more generally $X^{\mathsf{T}}\mathbb{E}\epsilon =0$ suffices. Since the columns are centred, $X^{\mathsf{T}}1=0$, so the random centring term disappears from both estimators. ThusLet $A=(S+\lambda I{)}^{-1}$. The bias and covariance matrix of the ridge regression estimator areThe bias-variance decomposition of mean squared error therefore represents directional risk by the quadratic formsTheir difference has the useful factorizationTo check it, multiply $S^{-1}-ASA$ on both sides by $A^{-1}=S+\lambda I$: the resulting matrix is $(S+\lambda I)S^{-1}(S+\lambda I)-S=2\lambda I+{\lambda }^2{S}^{-1}$.

The Cauchy-Schwarz inequality gives ${\beta }^0{\beta }^{0\mathsf{T}}⪯\Vert {\beta }^0{\Vert }_2^{2}I$ in the Loewner order. Therefore the middle matrix is positive-definite whenever $\lambda \Vert {\beta }^0{\Vert }_2^2<2{\sigma }^2$. For example,works, independently of the eigenvalues of $S$. Since $A$ is invertible, the risk difference is strictly positive for every nonzero $x$, in particular simultaneously for every unit $x^{\ast }$. This proves uniform directional risk improvement by ridge regression. For the normalized-loss convention, take the displayed choice divided by $n$.

For the final assertion, fix any $\lambda >0$ and choose a unit eigenvector $v$ of $S$ with eigenvalue $d>0$. Put $x^{\ast }=v$ and ${\beta }^0=Bv$. Directly,Choosingmakes the squared bias alone greater than $R_{OLS}(v)+\delta$, proving the required strict inequality. Thus unbounded directional risk of fixed ridge shrinkage is compatible with the existence of a beneficial signal-dependent small penalty: no fixed positive penalty uniformly improves risk over all possible signals.

The mean assumption matters if the printed covariance condition is read in isolation. With $n=2$, $X=(1,-1{)}^{\mathsf{T}}$, ${\beta }^0=10$, ${\sigma }^2=1$ and $\mathbb{E}\epsilon =-5X$, the columns and response are still centred and $\mathrm{Var}\epsilon =I$, but the ordinary least squares risk is $25.5$. Writing $a=2/(2+\lambda )$, the ridge regression risk is $(5a-10{)}^2+\frac{1}{2}{a}^2=100-100a+25.5a^2>25.5$ for every $0<a<1$. Hence the standard zero-mean regression-error convention, or its stated orthogonality version, is necessary for the first risk comparison; covariance alone is insufficient.