Past exam of the mathematics course of the University of Cambridge / 2015 / iii / Paper 33 / 4

Use an unpenalized intercept in ridge regression. For centered predictor columns, the ridge regression optimization isDifferentiating with respect to $\alpha$ gives $\hat{\alpha }=\overline{Y}$. Put $Y_c=Y-\overline{Y}1$. Differentiation with respect to $\beta$ gives the ridge normal equationConsequently the closed-form ridge regression estimator isBecause $X^{T}1=0$, the numerator may also be written $X^{T}Y$. For $\lambda >0$ the inverse exists even with multicollinearity or deficient column rank; at $\lambda =0$ a unique ordinary least squares coefficient vector requires full rank. The quadratic penalty stabilizes nearly singular directions but generally does not set individual coefficients exactly to zero.

For uncentered original predictors, use $X_c=X-1{\overline{x}}^T$, apply the displayed estimator to $X_c,Y_c$, and recover $\hat{\alpha }=\overline{Y}-{\overline{x}}^T\hat{\beta }$. Any internal scaling used by lm.ridge must be undone to report coefficients on the original predictor scale. Multiplying the objective by a constant changes the numerical penalty convention unless $\lambda$ is rescaled as well.

The generalized cross-validation curve reaches its minimum at $\lambda =5$ among the supplied grid values. This balances improved conditioning against shrinkage bias using an estimate of prediction error, rather than choosing the penalty with the smallest training residual sum of squares. Nearby values have similar errors, so the plot does not establish a highly precise optimal penalty.

Reading the PDF table at $\lambda =5$ gives the fitted regression intercept and slopes:These table entries are necessary because the TeX stores the table only inside a figure. The ridge regression slopes are shrunk relative to the zero-penalty fit; their small nonzero values do not represent variable exclusion.

There is a minor source inconsistency: the prose says the predictors are centered, but the table's intercept varies with $\lambda$. With exactly centered predictor columns and an unpenalized intercept it would remain $\overline{Y}$. The numbers above faithfully report the printed table, while part (a) gives the centered formula and the general uncentered conversion. The table therefore reflects a different or incompletely described preprocessing convention.

An intercept-unpenalized Lasso estimator solves the constrained optimizationFor centered predictors, eliminate the intercept as in ridge regression and minimize $\Vert Y_c-X\beta {\Vert }_2^2$ under the same constraint. Equivalently, with a suitable tuning parameter $\lambda \ge 0$, use $\Vert Y-\alpha 1-X\beta {\Vert }_2^2+\lambda {\sum }_j|{\beta }_j|$. The constraint radius and penalty parameter are different parametrizations; larger radius permits less shrinkage.

The Lasso plot parametrizes the Lasso regularization path by the fraction of the maximum $l_1$ norm of the standardized slopes. Unlike the ridge quadratic penalty, the corners of the $l_1$ constraint can place some slopes exactly at zero, providing variable selection. The maximum norm refers to the path's unpenalized endpoint; centering and scaling conventions must match those used to construct that path.

At fraction $0.8$, the Lasso regularization path lies between the vertical lines marked steps 4 and 5. In that interval the first five predictors have nonzero slopes: $X_1,X_2,X_4,X_5$ are positive and $X_3$ is negative. The sixth predictor remains at zero until the later part of the path, beyond this fraction. Therefore the chosen Lasso model includesThe regression intercept is retained. In particular the negative $X_3$ trace has already left zero at fraction $0.8$; a small slope is not the same as a zero slope. The selected fraction came from ten-fold K-fold cross-validation, and is not a numerical value of the ridge penalty in the preceding parts.