Past exam of the mathematics course of the University of Cambridge / 2025 / iii / Paper 205 / 5

The ridge regression estimator minimizesIts gradient vanishes exactly when $(X^{T}X+\lambda I)\beta =X^{T}Y$. Since $\lambda >0$ makes this matrix positive definite,

Use a singular value decomposition $X=UD{V}^T$. ThenEach nonzero singular value $d$ contributes $d/(d^2+\lambda )\to d^{-1}$, while each zero one contributes zero. Thususing the Moore-Penrose inverse.

Put $Z=[X,W]$. Its ridgeless minimum-norm fit is $Z^T(Z{Z}^T{)}^{+}Y$, so its first $p$ coordinates areThe strong law of large numbers applied entrywise gives $W{W}^T\to \lambda I_n$ almost surely as $d\to \infty$. Continuity of inversion then yieldsalmost surely, where the middle equality is the push-through identity.

Because $x_{\ast }\sim N_p(0,I)$ is independent, conditional prediction risk equals $\mathbb{E}(\Vert {\hat{\beta }}_{\lambda }-{\beta }^0{\Vert }_2^2\mid X)$. With $\hat{\Sigma }=X^{T}X/n$ and $\lambda =n\ell$,The noise term has conditional mean zero and covariance $\frac{{\sigma }^2}{n}(\hat{\Sigma }+\ell I{)}^{-1}\hat{\Sigma }(\hat{\Sigma }+\ell I{)}^{-1}$. Taking squared norms proves

Apply the stated deterministic equivalent for the squared resolvent with ${\Theta }_p={\beta }^0({\beta }^0{)}^T/r^2$. The bias term tends to $-{\ell }^2{r}^2{m}^{\prime }(\ell )$. For the variance useNormalized traces and $p/n\to \gamma$ therefore giveThus the conclusion printed in the paper also has both signs involving $m^{\prime }$ reversed. The hypotheses as printed imply the formula above; they cannot imply the requested one because $(\hat{\Sigma }+\ell I{)}^{-2}$ is positive definite while $m^{\prime }(\ell )\le 0$ for a resolvent limit.