 Solution

With the same squared-error normalization as in Question 2, the ridge estimator solvesThe unpenalized intercept in ridge regression is $\hat{a}=\overline{Y}$ because the columns of $X$ are centered. Differentiating the remaining quadratic objective gives $(X^{T}X+n\lambda I_p){\hat{\beta }}_{\lambda }^R=X^T{Y}_c$. Since $\lambda >0$, the left matrix is positive definite even when $X$ is rank-deficient. Thus the closed-form ridge regression estimator isThe second equality again uses centering. If the penalty convention is $\lambda \Vert b{\Vert }_2^2/2$ with an unnormalized squared-error term $\Vert Y_c-Xb{\Vert }_2^2/2$, replace $n\lambda$ by $\lambda$ throughout; the limiting assertion is unchanged.

For the full-column-rank case, take a singular value decomposition $X=UD{V}^T$ with positive singular values $d_1,\dots ,d_p$. The ordinary least squares estimator, which is the slope maximum-likelihood estimator in this normal linear model, and the ridge regression estimator are respectivelyConsequently principal-component shrinkage by ridge regression multiplies the least-squares coordinate in direction $v_j$ by $d_j^2/(d_j^2+n\lambda )$. Ridge shrinks most strongly in directions with the smallest singular values. These are combinations of predictors that the data distinguish least well.

Near multicollinearity creates small singular values, and the factor $1/d_j$ in ordinary least squares greatly amplifies noise in those directions. Its coefficient variance there is ${\sigma }^2/d_j^2$, while the ridge variance iswhich is smaller. The price is bias: the expectation of the ridge coordinate is $d_j^2/(d_j^2+n\lambda )$ times the corresponding true coefficient. This bias-variance tradeoff can substantially reduce estimation and prediction error when the poorly identified directions do not contain an excessively large signal. It is not a guarantee of improvement for every coefficient vector. Adding $n\lambda$ to every eigenvalue also improves the condition number of the normal matrix and makes numerical inversion more stable.

To handle every rank and every relation between $n$ and $p$, put $G=X^{T}X$ and use its spectral theorem for real symmetric matrices. Choose an orthonormal eigenbasis $v_1,\dots ,v_p$ with eigenvalues ${\gamma }_j\ge 0$, and put $z=X^{T}Y$. ThenIf ${\gamma }_j=0$, then $\Vert X{v}_j{\Vert }_2^2=v_j^{T}G{v}_j=0$, so $X{v}_j=0$ and $v_j^{T}z=(X{v}_j{)}^{T}Y=0$. Thus these terms are exactly zero for every positive $\lambda$; there is no divergent component in the null space. Each remaining term converges to $(v_j^{T}z/{\gamma }_j)v_j$. By the definition of the Moore-Penrose pseudoinverse,and thereforeThis proves the vanishing-penalty ridge limit without an invertibility assumption. The limit is the minimum-norm least-squares solution: every other least-squares solution differs by a vector in $\ker X$, orthogonal to the displayed solution, and therefore has at least as large a Euclidean norm.