= Solution
For an $n\times p$ <design matrix>, distinguish the coefficient vector from the fitted-response vector. The <ridge regression> objective and its solution are
$$
\boxed{Q_\lambda(\beta)=\|Y-X\beta\|_2^2+\lambda\|\beta\|_2^2,\qquad \widehat\beta_\lambda=(X^TX+\lambda I_p)^{-1}X^TY.}
$$
The <gradient> is $2(X^TX+\lambda I_p)\beta-2X^TY$. For $\lambda>0$, $X^TX+\lambda I_p$ is a <positive-definite matrix>, so this is the unique minimizer even if $X$ is rank deficient. The expression with an additional leading $X$ is instead the $n$-vector of <fitted values>,
$$
\widehat Y_\lambda=X\widehat\beta_\lambda=X(X^TX+\lambda I_p)^{-1}X^TY.
$$
This distinction corrects the dimensional mismatch in the printed coefficient-estimator notation.
The quadratic penalty discourages large coefficients, exchanging some bias for lower variance. If $X=UDV^T$ is a reduced <singular value decomposition>, with positive singular values $d_k$, then
$$
\widehat\beta_\lambda=V\operatorname{diag}\left(\frac{d_k}{d_k^2+\lambda}\right)U^TY,\qquad \widehat Y_\lambda=U\operatorname{diag}\left(\frac{d_k^2}{d_k^2+\lambda}\right)U^TY.
$$
The <principal-component shrinkage by ridge regression> therefore shrinks directions with small $d_k^2$ most strongly. These are precisely the poorly determined directions created by <multicollinearity>, where ordinary least-squares coefficients can have large variance or fail to be uniquely determined. A positive penalty stabilizes their inversion and sets coefficients in the null space to zero. It does not make the predictors independent or provide a universal risk improvement for every signal. Because the penalty depends on coefficient units, predictors are ordinarily standardized before choosing $\lambda$ by <cross-validation>. An <unpenalized intercept in ridge regression> is handled separately by centering; the displayed objective either has no intercept or applies the penalty to every component included in $\beta$.
Back to article page
