Past exam of the mathematics course of the University of Cambridge / 2018 / iii / Paper 218 / 6 / d

Let $\lambda$ denote the effective penalty in the Gaussian squared-error objective. With the loss normalization used in the glmnet documentation, alpha zero gives ridge regression:The intercept is fitted separately and unpenalized. The constant column supplied by model.matrix is not an additional penalized slope; glmnet excludes zero-variance predictor columns. Centering and the stated predictor norms make the two slope columns already standardized under the divisor-$n$ convention.

Let $X_s$ consist of those two columns, $z=y-\bar{y}1$, and $G=X_s^T{X}_s$. Minimizing over the intercept gives ${\hat{\beta }}_0=\bar{y}$, and the closed-form ridge regression estimator under this normalization isThus the right-hand limit in the plotted log-$\lambda$ coordinate is $\lambda \to \infty$, givingThe left-hand limit is $\lambda \to 0$, giving the unpenalized full-model slopesThe plotted curves are these two slopes. If the objective is instead written as unnormalized RSS plus a quadratic penalty, its penalty coefficient is $n\lambda$, not $\lambda$.

There is also a software response-scale convention to distinguish from this mathematical parameterization. The Gaussian fitting engine divides a supplied penalty by the response standard deviation $s_y=\sqrt{n^{-1}{\sum }_i(y_i-\bar{y}{)}^2}$ while fitting a standardized response, then rescales the coefficients. For pure ridge with these already standardized predictors, this corresponds to effective $\lambda ={\lambda }_{\mathrm{API}}/s_y$ in the displayed objective. Such a fixed positive rescaling of the horizontal parameter for the observed dataset does not change either asymptote. The fixed-penalty variance formulas below refer to $\lambda$ in the displayed mathematical objective.