= Ridge regression
{title2=$\operatorname{RSS}+\lambda\|\beta\|_2^2$}
{wiki}
= Ridge estimator
{synonym}
<Ridge regression> fits a <linear regression> model while penalizing the size of its slope <regression coefficients>. For a fixed $n\times p$ <design matrix> $X$ containing the slope predictors, observed responses $Y$, and an unpenalized <regression intercept> $\alpha$, the objective is
$$
(\widehat\alpha_\lambda,\widehat\beta_\lambda)=\underset{\alpha\in\mathbb R,\,\beta\in\mathbb R^p}{\operatorname{argmin}}\left\{\|Y-\alpha\mathbf1-X\beta\|_2^2+\lambda\|\beta\|_2^2\right\},\qquad\lambda\geq0.
$$
Here $\mathbf1$ is the all-ones <vector>, and $\|\beta\|_2^2=\sum_j\beta_j^2$ is the <squared Euclidean norm> of the slopes alone. The first term is the <residual sum of squares>; the second implements <regularization>. No random-error assumption is needed to define this fit. The convention is a summed <squared-error loss>: writing the objective as $n^{-1}\operatorname{RSS}+\lambda_{\mathrm{avg}}\|\beta\|_2^2$ gives the same fit when $\lambda=n\lambda_{\mathrm{avg}}$.<Data Mining: 36-462 and 36-662/Modern regression 1: Ridge regression>[slides 5 and 7: summed squared-error objective and unpenalized intercept]{ref}
For an exact example, reuse the constructed pairs $(x_i,Y_i)=(0,1),(1,2),(2,2),(3,4),(4,5)$ from <linear regression>. Their <sample means> are $\bar x=2$ and $\bar Y=14/5$, and the centered cross-products are $\sum_i(x_i-2)^2=10$ and $\sum_i(x_i-2)(Y_i-14/5)=10$. Thus
$$
\widehat\beta_\lambda=\frac{10}{10+\lambda},\qquad\widehat\alpha_\lambda=\frac{14}{5}-2\widehat\beta_\lambda,\qquad\widehat Y_i=\frac{14}{5}+\widehat\beta_\lambda(x_i-2).
$$
The <ordinary least squares> line has slope $1$ and <residual sum of squares> $4/5$. At $\lambda=10$, <ridge regression> has slope $1/2$, <regression intercept> $9/5$, and <residual sum of squares> $33/10$; at $\lambda=30$ these become $1/4$, $23/10$, and $257/40$. Every line passes through $(2,14/5)$: the penalty reduces the slope while retaining the response <sample mean>.
\Image[/ridge-regression-centered-line-fits.png]
{title=Exact synthetic ridge fits flatten around the sample mean as training RSS increases}
The computational benefit is visible after <predictor centering>. Write $X_c$ for the <design matrix> with each column's <sample mean> subtracted, and $Y_c=Y-\bar Y\mathbf1$. The <unpenalized intercept in ridge regression> gives $\widehat\alpha_\lambda=\bar Y-\bar x^T\widehat\beta_\lambda$, and the <closed-form ridge regression estimator> solves
$$
(X_c^TX_c+\lambda I_p)\widehat\beta_\lambda=X_c^TY_c.
$$
For any nonzero <vector> $v$ and $\lambda>0$, $v^T(X_c^TX_c+\lambda I_p)v=\|X_cv\|_2^2+\lambda\|v\|_2^2>0$. This <positive-definite matrix> is therefore invertible even when $X_c$ lacks <full column rank>, including repeated columns or $p\geq n$. The unique penalized slopes resolve the ambiguity of <rank-deficient ordinary least squares>; they do not supply evidence about slope directions absent from the data. This makes <ridge regression> useful for <multicollinearity>.<Advanced Topics in Statistical Learning, Spring 2023/High-Dimensional Regression: Ridge>[p. 1, equations (1)–(2): uniqueness for positive penalty regardless of design rank]{ref}
<Predictor centering> and scaling serve different purposes. Centering handles the free <regression intercept>. Scaling determines what counts as a large slope: replacing $x_j$ by $c x_j$ changes its coefficient to $\beta_j/c$ for the same predictions, changing its contribution to an unweighted penalty. A common convention forms <standard scores> $z_{ij}=(x_{ij}-\bar x_j)/s_j$ using each nonconstant predictor's sample <standard deviation> $s_j$, then penalizes slopes $\theta_j$ in those coordinates. Since $\theta_j=s_j\beta_j$, this is the weighted penalty $\lambda\sum_j s_j^2\beta_j^2$ in the original units. Constant columns have $s_j=0$ and must be removed before this scaling. Scaling is a modelling choice; equal penalties in original units and in standardized units define different fits.<Data Mining: 36-462 and 36-662/Modern regression 1: Ridge regression>[slide 7: center predictors, leave the intercept free and scale predictors to sample variance one]{ref}
The <singular value decomposition> explains the shrinkage. If $X_c=UDV^T$ is the compact decomposition using the $r$ positive <singular values> $d_j$, with columns $u_j$ of $U$, then
$$
\widehat Y_\lambda=\bar Y\mathbf1+\sum_{j=1}^r\frac{d_j^2}{d_j^2+\lambda}\,u_j u_j^TY_c.
$$
The <principal-component shrinkage by ridge regression> factors are between $0$ and $1$: directions with small <singular values> are damped most. As $\lambda\to\infty$, the <fitted values> approach $\bar Y\mathbf1$; the <vanishing-penalty ridge limit> recovers the <ordinary least squares> fitted vector as $\lambda\downarrow0$, even if the slope estimates at zero penalty are nonunique. The <covariance and bias of a ridge regression estimator> describes the resulting <bias-variance tradeoff>. A penalty can reduce <variance of an estimator> while introducing <bias of an estimator>; the <unbounded directional risk of fixed ridge shrinkage> rules out a uniform improvement for a fixed positive penalty over all possible signals.<Advanced Topics in Statistical Learning, Spring 2023/High-Dimensional Regression: Ridge>[pp. 1–2: shrinkage of fitted principal-component directions]{ref}
For a second exact construction, choose a centered <orthonormal set> of three <vectors>,
$$
u_1=\tfrac12(1,-1,1,-1)^T,\qquad u_2=\tfrac12(1,1,-1,-1)^T,\qquad u_3=\tfrac12(1,-1,-1,1)^T,
$$
and set $X=(3u_1,u_2)$ and $Y=2\mathbf1+3u_1+u_2+\tfrac12u_3$. The squared <singular values> are $9$ and $1$. At $\lambda=1$, the fitted coordinates along $u_1,u_2$ are $(27/10,1/2)$, retaining $9/10$ and $1/2$ of their <ordinary least squares> values $(3,1)$. At $\lambda=9$ they become $(3/2,1/10)$, retaining $1/2$ and $1/10$. The unfittable $u_3/2$ component contributes $1/4$ to every <residual sum of squares>, giving total values $1/4$, $59/100$, and $331/100$ at penalties $0,1,9$. These are constructed fits, not measured prediction results.
\Image[/ridge-regression-singular-direction-shrinkage.png]
{title=Exact synthetic ridge shrinkage retains the strong singular direction longer than the weak one}
The penalty $\lambda$ is a <hyperparameter>, so fitting slopes at a fixed value and selecting that value are separate tasks. Use a <validation set> or <K-fold cross-validation> to compare candidate penalties by held-out <squared-error loss>. In each split, compute predictor <sample means> and scales from that split's <training data>, apply the same transformation to its <validation set>, and refit the slopes; using held-out information in preprocessing introduces <data leakage>. Minimum training <residual sum of squares> cannot select useful shrinkage: <ordinary least squares> already attains the smallest possible value.<Data Mining: 36-462 and 36-662/Model selection and validation 1: Cross-validation>[slides 5–8 and 11–14: tuning for prediction, training error and refitting on each training fold]{ref} After <model selection>, refit on all <training data> with the selected penalty. A separate <test set> or <nested cross-validation> assesses the entire selection procedure; a low selected validation score alone is no performance guarantee.<On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation by Gavin C. Cawley and others>[pp. 2079–2080: selection as part of fitting and nested assessment]{ref}
Back to article page
