Past exam of the mathematics course of the University of Cambridge / 2023 / iii / Paper 218 / 5

For fixed ${\lambda }_2$, letting ${\lambda }_1\downarrow 0$ gives ridge regression, including ordinary least squares when ${\lambda }_2=0$; letting ${\lambda }_1\to \infty$ forces every coefficient to zero. For fixed ${\lambda }_1$, letting ${\lambda }_2\downarrow 0$ gives the Lasso. Letting both penalties vanish gives an ordinary least squares solution, unique when $X$ has full column rank and otherwise potentially nonunique or path-dependent.

When ${\lambda }_2>0$, the term ${\lambda }_2\Vert \beta {\Vert }_2^2$ is strictly convex. Its sum with the convex squared loss and ${\ell }^1$ penalty is strictly convex and coercive, so the elastic net solution exists and is unique.

For coordinate $j$, hold all other coefficients fixed and form the partial residualThe one-coordinate coordinate descent problem isIts exact update isCyclically update coordinates and their residuals until the objective or coefficients converge. Convexity makes every limit point a global minimizer, and ${\lambda }_2>0$ makes it the unique minimizer.

When $X^{T}X=I_p$, expanding the objective separates it by coordinates:The soft thresholding solution is therefore

Model m1 is ridge regression, while m2 is the Lasso. Ridge shrinks but normally retains every coefficient; the Lasso's ${\ell }^1$ penalty sets many coefficients exactly to zero. Model m3 combines sparsity with the grouping effect of the elastic net: correlated predictors tend to enter together and receive more similar coefficients.

Accordingly, m3 keeps the weak variables age, lcp, and gleason at zero as m2 does, but retains lweight, lbph, svi, and pgg45 as the ridge fit does. The correlation $0.54$ between svi and lcavol explains why m3 keeps both with substantial coefficients, whereas m2 selects lcavol and discards svi. This lies between the dense ridge behavior and the more aggressively sparse Lasso behavior.

Choose $({\lambda }_1,{\lambda }_2)$ on a grid by K-fold cross-validation, comparing the same held-out prediction loss and optionally applying the one-standard-error rule for a simpler model. A genuinely untouched test set can then estimate final prediction error.

Ordinary model-based intervals after selecting nonzero coefficients ignore selection and are generally invalid. Valid approaches include Debiased Lasso or a selective-inference procedure under its assumptions, sample splitting followed by an unpenalized refit and inference on the independent half, or a bootstrap that repeats both tuning and fitting and is interpreted with care near the nonsmooth zero threshold.