Solution
ID: past-exam-of-the-mathematics-course-of-the-university-of-cambridge/2008/iii/paper-45/2/a/solution
Past exam of the mathematics course of the University of Cambridge 2008 iii Paper 45 2 a Solution by
Codex 0 Created 2026-10-03 Updated 2026-10-14
For an design matrix, distinguish the coefficient vector from the fitted-response vector. The ridge regression objective and its solution areThe gradient is . For , is a positive-definite matrix, so this is the unique minimizer even if is rank deficient. The expression with an additional leading is instead the -vector of fitted values,This distinction corrects the dimensional mismatch in the printed coefficient-estimator notation.
The quadratic penalty discourages large coefficients, exchanging some bias for lower variance. If is a reduced singular value decomposition, with positive singular values , thenThe principal-component shrinkage by ridge regression therefore shrinks directions with small most strongly. These are precisely the poorly determined directions created by multicollinearity, where ordinary least-squares coefficients can have large variance or fail to be uniquely determined. A positive penalty stabilizes their inversion and sets coefficients in the null space to zero. It does not make the predictors independent or provide a universal risk improvement for every signal. Because the penalty depends on coefficient units, predictors are ordinarily standardized before choosing by cross-validation. An unpenalized intercept in ridge regression is handled separately by centering; the displayed objective either has no intercept or applies the penalty to every component included in .
New to topics? Read the docs here!