Past exam of the mathematics course of the University of Cambridge / 2025 / iii / Paper 218 / 2 / b / Solution

Put $z={\hat{\beta }}_{\mathrm{OLS}}=X^{T}Y/n$. Under $X^{T}X=n{I}_p$, the objective separates by coordinates. Completing the square and applying the soft-thresholding operator $S(u,t)=\mathrm{sign}(u)(|u|-t{)}_{+}$ givesFor a fixed $z_j$, its magnitude lies between the ridge endpoint $|z_j|/(1+\lambda )$ and the lasso endpoint $(|z_j|-\lambda {)}_{+}$; this follows directly on the two intervals $\lambda \alpha \ge |z_j|$ and $\lambda \alpha <|z_j|$ by cross-multiplication. Thus the stated endpoint inequality holds.

For $alpha>0$ and $z_j\ne 0$, the coordinate first vanishes when the soft threshold reaches $|z_j|$, soThis is strictly decreasing in $\alpha$, and it diverges to infinity as $\alpha \downarrow 0$. This agrees with the fact that pure ridge shrinkage does not set a nonzero coordinate exactly to zero at any finite penalty.