Ridge regression ($\mathrm{RSS}+\lambda \Vert \beta {\Vert }_2^2$)

Ridge regression fits a linear regression model while penalizing the size of its slope regression coefficients. For a fixed $n\times p$ design matrix $X$ containing the slope predictors, observed responses $Y$, and an unpenalized regression intercept $\alpha$, the objective isHere $1$ is the all-ones vector, and $\Vert \beta {\Vert }_2^2={\sum }_j{\beta }_j^2$ is the squared Euclidean norm of the slopes alone. The first term is the residual sum of squares; the second implements regularization. No random-error assumption is needed to define this fit. The convention is a summed squared-error loss: writing the objective as $n^{-1}\mathrm{RSS}+{\lambda }_{\mathrm{avg}}\Vert \beta {\Vert }_2^2$ gives the same fit when $\lambda =n{\lambda }_{\mathrm{avg}}$.^[ref]

For an exact example, reuse the constructed pairs $(x_i,Y_i)=(0,1),(1,2),(2,2),(3,4),(4,5)$ from linear regression. Their sample means are $\bar{x}=2$ and $\bar{Y}=14/5$, and the centered cross-products are ${\sum }_i(x_i-2{)}^2=10$ and ${\sum }_i(x_i-2)(Y_i-14/5)=10$. ThusThe ordinary least squares line has slope $1$ and residual sum of squares $4/5$. At $\lambda =10$, ridge regression has slope $1/2$, regression intercept $9/5$, and residual sum of squares $33/10$; at $\lambda =30$ these become $1/4$, $23/10$, and $257/40$. Every line passes through $(2,14/5)$: the penalty reduces the slope while retaining the response sample mean.

The computational benefit is visible after predictor centering. Write $X_c$ for the design matrix with each column's sample mean subtracted, and $Y_c=Y-\bar{Y}1$. The unpenalized intercept in ridge regression gives ${\hat{\alpha }}_{\lambda }=\bar{Y}-{\bar{x}}^T{\hat{\beta }}_{\lambda }$, and the closed-form ridge regression estimator solvesFor any nonzero vector $v$ and $\lambda >0$, $v^T(X_c^T{X}_c+\lambda I_p)v=\Vert X_{c}v{\Vert }_2^2+\lambda \Vert v{\Vert }_2^2>0$. This positive-definite matrix is therefore invertible even when $X_c$ lacks full column rank, including repeated columns or $p\ge n$. The unique penalized slopes resolve the ambiguity of rank-deficient ordinary least squares; they do not supply evidence about slope directions absent from the data. This makes ridge regression useful for multicollinearity.^[ref]

Predictor centering and scaling serve different purposes. Centering handles the free regression intercept. Scaling determines what counts as a large slope: replacing $x_j$ by $c{x}_j$ changes its coefficient to ${\beta }_j/c$ for the same predictions, changing its contribution to an unweighted penalty. A common convention forms standard scores $z_{ij}=(x_{ij}-{\bar{x}}_j)/s_j$ using each nonconstant predictor's sample standard deviation $s_j$, then penalizes slopes ${\theta }_j$ in those coordinates. Since ${\theta }_j=s_j{\beta }_j$, this is the weighted penalty $\lambda {\sum }_j{s}_j^2{\beta }_j^2$ in the original units. Constant columns have $s_j=0$ and must be removed before this scaling. Scaling is a modelling choice; equal penalties in original units and in standardized units define different fits.^[ref]

The singular value decomposition explains the shrinkage. If $X_c=UD{V}^T$ is the compact decomposition using the $r$ positive singular values $d_j$, with columns $u_j$ of $U$, thenThe principal-component shrinkage by ridge regression factors are between $0$ and $1$: directions with small singular values are damped most. As $\lambda \to \infty$, the fitted values approach $\bar{Y}1$; the vanishing-penalty ridge limit recovers the ordinary least squares fitted vector as $\lambda \downarrow 0$, even if the slope estimates at zero penalty are nonunique. The covariance and bias of a ridge regression estimator describes the resulting bias-variance tradeoff. A penalty can reduce variance of an estimator while introducing bias of an estimator; the unbounded directional risk of fixed ridge shrinkage rules out a uniform improvement for a fixed positive penalty over all possible signals.^[ref]

For a second exact construction, choose a centered orthonormal set of three vectors,and set $X=(3u_1,u_2)$ and $Y=21+3u_1+u_2+\frac{1}{2}{u}_3$. The squared singular values are $9$ and $1$. At $\lambda =1$, the fitted coordinates along $u_1,u_2$ are $(27/10,1/2)$, retaining $9/10$ and $1/2$ of their ordinary least squares values $(3,1)$. At $\lambda =9$ they become $(3/2,1/10)$, retaining $1/2$ and $1/10$. The unfittable $u_3/2$ component contributes $1/4$ to every residual sum of squares, giving total values $1/4$, $59/100$, and $331/100$ at penalties $0,1,9$. These are constructed fits, not measured prediction results.

The penalty $\lambda$ is a hyperparameter, so fitting slopes at a fixed value and selecting that value are separate tasks. Use a validation set or K-fold cross-validation to compare candidate penalties by held-out squared-error loss. In each split, compute predictor sample means and scales from that split's training data, apply the same transformation to its validation set, and refit the slopes; using held-out information in preprocessing introduces data leakage. Minimum training residual sum of squares cannot select useful shrinkage: ordinary least squares already attains the smallest possible value.^[ref] After model selection, refit on all training data with the selected penalty. A separate test set or nested cross-validation assesses the entire selection procedure; a low selected validation score alone is no performance guarantee.^[ref]

For likelihood $Y\mid \beta \sim N(X\beta ,{\sigma }^{2}I)$ and independent prior $\beta \sim N(0,{\tau }^{2}I)$, the posterior distribution is Gaussian with covariance $V$ as displayed and mean $(X^{T}X+\lambda I{)}^{-1}{X}^{T}Y$. Its mean and mode therefore coincide with the ridge regression coefficient estimate. A Gaussian prior alone does not imply this conclusion without the stated likelihood.

// Target: foundations-of-mathematics.bigb

For fixed positive ridge regression penalty, choose a unit eigenvector $v$ of $X^{\mathsf{T}}X$ with eigenvalue $d>0$, and signal ${\beta }^0=Bv$. The directional ridge risk is $({\lambda }^2{B}^2+{\sigma }^{2}d)/(d+\lambda {)}^2$, whereas ordinary least squares has risk ${\sigma }^2/d$. Letting $|B|$ grow makes the ridge squared bias arbitrarily large.

For a full-rank centred design, unscaled ridge regression and zero-mean errors with covariance ${\sigma }^{2}I$, put $S=X^{\mathsf{T}}X$ and $A=(S+\lambda I{)}^{-1}$. The ordinary-least-squares risk matrix minus the ridge risk matrix is $\lambda A(2{\sigma }^{2}I+\lambda {\sigma }^2{S}^{-1}-\lambda {\beta }^0{\beta }^{0\mathsf{T}})A$. It is positive definite whenever $\lambda \Vert {\beta }^0{\Vert }_2^2<2{\sigma }^2$, including every positive $\lambda$ when ${\beta }^0=0$. Thus one signal-dependent small penalty improves every nonzero-direction mean squared error.

In ridge regression, penalize slopes but ordinarily leave the regression intercept free. Centering gives $\hat{\beta }=(X_c^T{X}_c+\lambda I{)}^{-1}{X}_c^T{Y}_c$ and $\hat{\alpha }=\overline{Y}-{\overline{x}}^T\hat{\beta }$. An exactly centered predictor matrix has $\hat{\alpha }=\overline{Y}$ for every penalty. A table with a penalty-dependent intercept therefore cannot arise from exactly centered columns under this convention.

For a fixed centered design matrix $X$, independent errors of variance ${\sigma }^2$, and objective $\Vert Y-X\beta {\Vert }^2+a\Vert \beta {\Vert }^2$, put $G=X^{T}X$ and $M=G+aI$. The ridge regression estimator hasThus its standard error alone does not justify centering a confidence interval for $\beta$ at the biased estimator. A penalty chosen from the responses also changes its sampling distribution.

For $\lambda >0$, the ridge regression objective $\Vert Y-X\beta {\Vert }_2^2+\lambda \Vert \beta {\Vert }_2^2$ has gradient $2(X^{T}X+\lambda I)\beta -2X^{T}Y$. The matrix $X^{T}X+\lambda I$ is positive definite, so the unique minimizer is ${\hat{\beta }}_{\lambda }=(X^{T}X+\lambda I{)}^{-1}{X}^{T}Y$.

In an eigenbasis of $G=X^{T}X$, ridge regression divides the corresponding component of $X^{T}Y$ by $\gamma +\lambda$. If $\gamma =0$, its eigenvector $v$ satisfies $Xv=0$, and hence $v^T{X}^{T}Y=0$ exactly. Only positive eigenvalues contribute to the limit, giving the Moore-Penrose inverse expression for the minimum-norm least-squares solution. No full-rank or sample-size assumption is needed.

For any real design matrix $X$ and $\lambda >0$, $(X^{T}X+\lambda I{)}^{-1}{X}^T=X^T(X{X}^T+\lambda I{)}^{-1}$. Multiplication by $X$ identifies the primal ridge regression fitted values with $K(K+\lambda I{)}^{-1}Y$, $K=X{X}^T$, the kernel-ridge hat matrix for the linear kernel. No rank assumption is needed.

For $X^{T}X=V\Lambda V^T$, ridge regression multiplies the fitted response along the $i$th normalized sample principal component by the shrinkage factor ${\Lambda }_{ii}/({\Lambda }_{ii}+\lambda )$. Directions with variance much larger than $\lambda$ are nearly retained, while directions with variance much smaller than $\lambda$ are nearly removed.