Past exam of the mathematics course of the University of Cambridge / 2015 / iii / Paper 33 / 4 / a / Solution

Use an unpenalized intercept in ridge regression. For centered predictor columns, the ridge regression optimization isDifferentiating with respect to $\alpha$ gives $\hat{\alpha }=\overline{Y}$. Put $Y_c=Y-\overline{Y}1$. Differentiation with respect to $\beta$ gives the ridge normal equationConsequently the closed-form ridge regression estimator isBecause $X^{T}1=0$, the numerator may also be written $X^{T}Y$. For $\lambda >0$ the inverse exists even with multicollinearity or deficient column rank; at $\lambda =0$ a unique ordinary least squares coefficient vector requires full rank. The quadratic penalty stabilizes nearly singular directions but generally does not set individual coefficients exactly to zero.

For uncentered original predictors, use $X_c=X-1{\overline{x}}^T$, apply the displayed estimator to $X_c,Y_c$, and recover $\hat{\alpha }=\overline{Y}-{\overline{x}}^T\hat{\beta }$. Any internal scaling used by lm.ridge must be undone to report coefficients on the original predictor scale. Multiplying the objective by a constant changes the numerical penalty convention unless $\lambda$ is rescaled as well.