Closed-form ridge regression estimator (${\hat{\beta }}_{\lambda }=(X^{T}X+\lambda I{)}^{-1}{X}^{T}Y$)

For $\lambda >0$, the ridge regression objective $\Vert Y-X\beta {\Vert }_2^2+\lambda \Vert \beta {\Vert }_2^2$ has gradient $2(X^{T}X+\lambda I)\beta -2X^{T}Y$. The matrix $X^{T}X+\lambda I$ is positive definite, so the unique minimizer is ${\hat{\beta }}_{\lambda }=(X^{T}X+\lambda I{)}^{-1}{X}^{T}Y$.

For $X^{T}X=V\Lambda V^T$, ridge regression multiplies the fitted response along the $i$th normalized sample principal component by the shrinkage factor ${\Lambda }_{ii}/({\Lambda }_{ii}+\lambda )$. Directions with variance much larger than $\lambda$ are nearly retained, while directions with variance much smaller than $\lambda$ are nearly removed.