Past exam of the mathematics course of the University of Cambridge / 2008 / iii / Paper 45 / 2

For an $n\times p$ design matrix, distinguish the coefficient vector from the fitted-response vector. The ridge regression objective and its solution areThe gradient is $2(X^{T}X+\lambda I_p)\beta -2X^{T}Y$. For $\lambda >0$, $X^{T}X+\lambda I_p$ is a positive-definite matrix, so this is the unique minimizer even if $X$ is rank deficient. The expression with an additional leading $X$ is instead the $n$-vector of fitted values,This distinction corrects the dimensional mismatch in the printed coefficient-estimator notation.

The quadratic penalty discourages large coefficients, exchanging some bias for lower variance. If $X=UD{V}^T$ is a reduced singular value decomposition, with positive singular values $d_k$, thenThe principal-component shrinkage by ridge regression therefore shrinks directions with small $d_k^2$ most strongly. These are precisely the poorly determined directions created by multicollinearity, where ordinary least-squares coefficients can have large variance or fail to be uniquely determined. A positive penalty stabilizes their inversion and sets coefficients in the null space to zero. It does not make the predictors independent or provide a universal risk improvement for every signal. Because the penalty depends on coefficient units, predictors are ordinarily standardized before choosing $\lambda$ by cross-validation. An unpenalized intercept in ridge regression is handled separately by centering; the displayed objective either has no intercept or applies the penalty to every component included in $\beta$.

Use the Gaussian sampling model $Y\mid \beta \sim N(X\beta ,{\sigma }^2{I}_n)$ with known ${\sigma }^2>0$, and the independent prior $\beta \sim N(0,{\tau }^2{I}_p)$ with ${\tau }^2>0$. A Gaussian prior alone is not enough to imply the ridge result without this likelihood assumption. The posterior distribution is proportional toMaximizing it is equivalent to minimizing the ridge objective with $\lambda ={\sigma }^2/{\tau }^2$. More explicitly, complete the square. The posterior precision, covariance matrix and mean areConsequently the Gaussian posterior representation of ridge regression isA tighter prior, smaller ${\tau }^2$, implies stronger shrinkage; larger observation-noise variance at fixed prior scale also increases the penalty. The posterior covariance is $V$, not the frequentist sampling covariance of the ridge estimator, which is ${\sigma }^2(X^{T}X+\lambda I{)}^{-1}{X}^{T}X(X^{T}X+\lambda I{)}^{-1}$. If an intercept is to remain unpenalized, give only the slopes this proper Gaussian prior and center the design and responses.

The ridge regression penalty is quadratic, whereas the Lasso uses an absolute-value penalty. To make the constants explicit, compareBoth trade goodness of fit against coefficient size, and both require sensible predictor scaling and tuning. Ridge continuously shrinks coefficient directions and generally retains all variables. The Lasso's nonsmooth penalty can set coefficients exactly to zero, so it performs variable selection as well as shrinkage. When $X^{T}X=I$, let $z=X^{T}Y$. The distinction is especially transparent:The second formula is soft thresholding. For general designs, the Karush-Kuhn-Tucker conditions for the Lasso give $X_j^T(Y-X\hat{\beta })=t\mathrm{sign}({\hat{\beta }}_j)$ for a nonzero coefficient and $|X_j^T(Y-X\hat{\beta })|\le t$ for a zero coefficient. The latter interval permits exact zeros; the ridge equation instead is $X^T(Y-X\hat{\beta })=\lambda \hat{\beta }$.

Ridge is especially useful for stable prediction when many predictors carry small effects or are highly correlated. Its strictly convex penalty gives unique coefficients for $\lambda >0$ even in a rank-deficient design. It shares an effect among identical predictor columns: for a fixed coefficient sum the minimum quadratic penalty splits that sum equally. The Lasso can give a simpler sparse model when many effects are absent, but selection among strongly correlated predictors can be unstable. With identical columns, any same-sign split of a selected coefficient sum has the same absolute-value penalty and the same fitted values, so the coefficient vector need not be unique. This is a possible design degeneracy, not a claim that every Lasso fit is nonunique.

The penalties also encode different prior shapes: the Gaussian prior yields ridge, while independent double-exponential priors yield a Lasso posterior mode under Gaussian noise. A Lasso mode is not generally its posterior mean. Practically, sparse interpretability favors the Lasso, while stable distributed effects can favor ridge; prediction performance should be checked with cross-validation, and neither method universally dominates the other.