Past exam of the mathematics course of the University of Cambridge / 2025 / iii / Paper 218 / 2

The ridge regression estimator is the elastic net at $\alpha =0$, while the lasso regression estimator is the elastic net at $\alpha =1$.

Put $z={\hat{\beta }}_{\mathrm{OLS}}=X^{T}Y/n$. Under $X^{T}X=n{I}_p$, the objective separates by coordinates. Completing the square and applying the soft-thresholding operator $S(u,t)=\mathrm{sign}(u)(|u|-t{)}_{+}$ givesFor a fixed $z_j$, its magnitude lies between the ridge endpoint $|z_j|/(1+\lambda )$ and the lasso endpoint $(|z_j|-\lambda {)}_{+}$; this follows directly on the two intervals $\lambda \alpha \ge |z_j|$ and $\lambda \alpha <|z_j|$ by cross-multiplication. Thus the stated endpoint inequality holds.

For $alpha>0$ and $z_j\ne 0$, the coordinate first vanishes when the soft threshold reaches $|z_j|$, soThis is strictly decreasing in $\alpha$, and it diverges to infinity as $\alpha \downarrow 0$. This agrees with the fact that pure ridge shrinkage does not set a nonzero coordinate exactly to zero at any finite penalty.

Writing the horizontal coordinate as $u=\log \lambda$, so that $\lambda =e^u$, the two coefficient paths at $\alpha =1/2$ areEquivalently, as functions of $\lambda$, replace every $e^u$ by $\lambda$. The paths reach zero at $lambda=2.82$ and $lambda=0.82$, respectively.

Let $z={\hat{\beta }}_{\mathrm{OLS}}$ and $G=X^{T}X/n=\left(\begin{array}{cc}1& \rho \\ \rho & 1\end{array}\right)$. While both lasso coordinates are positive, the Karush-Kuhn-Tucker conditions giveHenceAssume without loss of generality that $z_1\le z_2$. After the first coordinate vanishes, the second remains active until $lambda=z_2+\rho z_1$. The zero vector satisfies the KKT conditions exactly when $lambda\ge \Vert Gz{\Vert }_{\infty }$; for $z_1\le z_2$ and $-1<\rho <1$, this norm is $z_2+\rho z_1$. Thereforeandwhich is independent of $\rho$.