Linear regression ($Y=X\beta +\epsilon$)

Linear regression models a response's conditional expectation as a linear combination of specified predictors. In matrix form,where the design matrix $X$ describes the observations, $\beta$ contains the regression coefficients, and $\epsilon$ contains the errors. The fitted values $X\hat{\beta }$ estimate the response means. The model is linear in its regression coefficients; its columns may include powers, interactions, or other transformed predictors, so polynomial regression can fit curved relationships using the same machinery.

For a concrete example, take five constructed observationsAn ordinary least squares fit with an intercept gives the line $\hat{y}=0.8+x$. Its regression residuals are $(0.2,0.2,-0.8,0.2,0.2)$ and its residual sum of squares is $0.8$. A one-unit increase in the predictor corresponds to a one-unit increase in the fitted response. This is a description of this fitted association; interpreting a change as a causal effect requires the assumptions of a causal inference problem.

The geometric reason for the fit is simple. The possible mean vectors $X\beta$ form the column space of $X$. Ordinary least squares chooses the orthogonal projection of the data vector onto that vector subspace, so the remaining regression residual is orthogonal to every permitted fitted direction. The least-squares normal equations express this orthogonality as $X^T(Y-X\hat{\beta })=0$. The data's explained component and unexplained component are therefore separated by the same geometry used in a Hilbert space.

The optimization definition makes sense for any observed data. Quantifying the uncertainty of the regression coefficients needs additional assumptions about the sampling and errors. With fixed $X$ of full column rank and mean-zero errors with covariance matrix ${\sigma }^{2}I$, ordinary least squares is unbiased and has covariance ${\sigma }^2(X^{T}X{)}^{-1}$; the Gauss-Markov theorem compares it with other linear unbiased estimators. A normal linear model additionally makes Student t-distribution intervals and tests exact in finite samples. Model adequacy and predictive assessment remain relevant even when these calculations are available.

Two historical settings help explain the subject's appeal. Adrien-Marie Legendre's Nouvelles méthodes pour la détermination des orbites des comètes (1805) includes an appendix on the method of least squares: observations give more equations than unknowns, and squared errors provide a criterion for reconciling them. Its astronomical setting shows the motivation for combining imperfect measurements. ^[ref] Francis Galton's Regression Towards Mediocrity in Hereditary Stature (1886) examines the association between parental and offspring heights and the tendency of group averages toward the population average. This is a historical use of the regression name; the modern linear model is a general statistical formulation. ^[ref]

Fit a quadratic mean at three temperatures $x_0,x_0\pm 1$, with equal replication $n$ at the off-center temperatures and any positive replication $m$ at the center. The three parameters fit the three group means exactly, giving the displayed estimates. Independent Gaussian errors make these estimates jointly normal. Their covariance is $\frac{1}{4}[\mathrm{Var}({\bar{Y}}_{+})-\mathrm{Var}({\bar{Y}}_{-})]=0$, so they are independent. Equal off-center replication is essential to this cancellation.

At symmetric design points, the columns $1$, $x$ and $x^2-\overline{x^2}$ are mutually orthogonal. Their squared lengths give the diagonal entries of the normal-equation matrix and hence separate least-squares coefficient variances. At the seven integer points from minus three to three, these squared lengths are $7,28,84$.

Suppose ${ϵ}_1,{ϵ}_2,\dots$ are independent and identically distributed with mean zero and finite variance ${\sigma }^2>0$. For deterministic weights $x_{n,i}$, put $S_n={\sum }_{i=1}^n{x}_{n,i}^2>0$. Ifthen the Lindeberg-Feller central limit theorem givesThis is the condition that no observation dominates the weighted error sum. In a centered linear regression with orthogonal explanatory columns, ${\hat{\beta }}_1-{\beta }_1={\sum }_i{x}_{n,i}{ϵ}_i/S_n$, so the result justifies a normal approximation with variance ${\sigma }^2/S_n$.

To verify the Lindeberg condition, write $a_{n,i}=x_{n,i}/\sqrt{S_n}$ and ${\delta }_n={\max }_i|a_{n,i}|\to 0$. For each $\eta >0$,The inequality uses ${\sum }_i{a}_{n,i}^2=1$, and the limit follows from the finite second moment. Without the maximum-weight condition, adding observations with zero explanatory value can leave the slope determined by a fixed number of nonnormal errors for every sample size.

A statistical model specifying a response as a linear combination of known regressor columns and an error. Common assumptions are $\mathbb{E}[\epsilon ]=0$ and $\mathrm{Cov}(\epsilon )={\sigma }^{2}I$, although a normal distribution is an additional assumption, not part of the displayed equation alone. Full column rank of the design matrix makes the coefficients identifiable. Statistical association does not by itself give a causal effect.

An exponential decay rate $R=R_0{e}^{-t/\tau }$ becomes a straight line in $x=t$ and $y=\ln (R/R_{\ast })$, where $R_{\ast }$ is a fixed reference rate. Constant small fractional rate errors become approximately constant vertical errors $s\simeq {\sigma }_R/R$ by propagation of uncertainty. With independent Gaussian log-rate errors, linear least squares gives $\hat{m}=S_{xy}/S_{xx}$ and $\hat{c}=\bar{y}-\hat{m}\bar{x}$.

The covariance matrix is obtained by applying the variance rules to these linear estimators:Transforming back gives $\hat{\tau }=-1/\hat{m}$, ${\sigma }_{\tau }\simeq {\sigma }_m/{\hat{m}}^2$, ${\hat{R}}_0=R_{\ast }{e}^{\hat{c}}$ and ${\sigma }_{R_0}\simeq {\hat{R}}_0{\sigma }_c$. If a candidate lifetime fixes the slope $m_a$, the intercept must be refitted as $c_a=\bar{y}-m_a\bar{x}$. The profiled objective obeysThis follows by separating the centred slope residual from the unrestricted regression residual, whose inner product with the design columns is zero by the normal equations for linear least squares. Known supplied measurement errors are not rescaled by the observed reduced chi-squared. This log-error model is an approximation for proportional errors, not an exact replacement for every raw-count likelihood.

Multiple linear regression models a scalar response by an intercept and several predictors. It is linear in the regression coefficients, even when predictors include transformed variables or interactions. In an observational study, adjustment can address measured confounding, but does not automatically identify a causal effect. Improvements in fitted coefficient of determination require external predictive assessment rather than being treated as proof of improved future prediction.

Minimizing $\Vert y-X\theta {\Vert }^2$ gives the displayed equations by differentiation. Equivalently, the regression residual $r=y-X\hat{\theta }$ satisfies $X^{T}r=0$: it is orthogonal to every column of the design matrix. If $X$ has full column rank, the minimizer is unique and the fitted values are the orthogonal projection of $y$ onto its column space. A rank-deficient design matrix can give multiple coefficient vectors, while the projected fitted vector is still unique.

For an intercept and one predictor, put $S_{xx}={\sum }_i(x_i-\bar{x}{)}^2$ and $S_{xy}={\sum }_i(x_i-\bar{x})(y_i-\bar{y})$. The equations give $\hat{\beta }=S_{xy}/S_{xx}$ and $\hat{\alpha }=\bar{y}-\hat{\beta }\bar{x}$, provided $S_{xx}>0$. In the five-point example in linear regression, $\bar{x}=2$, $\bar{y}=2.8$, $S_{xx}=10$ and $S_{xy}=10$, giving $\hat{\beta }=1$ and $\hat{\alpha }=0.8$.

The normal equations describe the mathematical optimum, but explicitly forming and inverting $X^{T}X$ can amplify numerical problems. A QR decomposition computes the same least-squares solution through orthogonal coordinates; a singular value decomposition also exposes nearly dependent columns. This connects statistical fitting to numerical linear algebra and the choice of a stable algorithm. ^[ref]

If the true conditional mean is ${\beta }_0+{\beta }_{1}X+{\beta }_{2}Z$, the population slope from regressing on $X$ alone is ${\beta }_1+{\beta }_2\mathrm{Cov}(X,Z)/\mathrm{Var}(X)$, assuming a mean-zero error uncorrelated with the predictors. The omitted predictor can steepen, flatten, or reverse the pooled association. The same identity holds for fitted coefficients and empirical covariances when comparing nested ordinary least-squares fits. Adjustment changes the comparison being described; a causal interpretation requires additional assumptions.

Replacing a numerical predictor $x$ by $x-c$ preserves the fitted mean space and slope in a linear regression containing an intercept. The intercept becomes the fitted mean at $x=c$. In simple linear regression, centering at the sample mean makes intercept and slope estimates uncorrelated under a normal linear model. This changes coefficient interpretation, not predictions.

Regression diagnostics check assumptions and influential observations in a fitted regression function. Residual-versus-fitted plots can reveal omitted mean structure or nonconstant variance; a quantile-quantile plot checks a specified error distribution. Regression leverage measures unusual predictor configurations, and Cook's distance combines leverage and residual size to measure coefficient sensitivity. Independence may require checking collection order and the study design in addition to residual plots.

An influential observation substantially changes a fitted model or its predictions when removed or perturbed. In a linear regression, influence depends jointly on residual size and regression leverage. Large predictor leverage with a small residual and a large residual at ordinary leverage can have different consequences. Regression diagnostics examine this sensitivity; influence is not synonymous with an invalid observation.

A regression outlier has a response unusually far from its fitted conditional mean. A standardized regression residual adjusts for regression leverage; an externally studentized residual also estimates noise variance with the observation deleted. Selecting the largest residual among many observations changes its reference distribution. An outlier need not be an influential observation, and unusual valid data should not be discarded solely for being unusual.

A piecewise linear regression-path algorithm which starts at zero, activates a predictor with maximal absolute residual score, and moves in a direction that decreases all active absolute scores equally until another predictor ties them. With $G=X^{\mathsf{T}}X/n$ and active correlation signs $s_A$, the coefficient direction is $G_{AA}^{-1}{s}_A$. Ordinary LAR does not drop a predictor when its coefficient crosses zero; sign compatibility of LAR and Lasso characterizes when its path also obeys the Lasso KKT conditions.

With signed active covariate columns $X_A^s=(s_j{x}_j{)}_{j\in A}$ and their positive-definite Gram matrix $G_A=(X_A^s{)}^T{X}_A^s$, the displayed direction has unit norm and $(X_A^s{)}^{T}u=\alpha 1$. Thus it makes equal angles with unit-length signed active covariates, and movement along it reduces their signed residual correlations at the common rate $\alpha$. The regression coefficient direction is $d_A=\mathrm{diag}(s_A)w$, with inactive components zero.

If every nonzero active coefficient has the sign of its residual score along the least angle regression path, the LAR active correlation invariant and inactive bounds give the Karush-Kuhn-Tucker conditions for the Lasso. A unique Lasso solution then equals the LAR path. Zero coefficients use a subgradient interval; literal sign equality with sign(0)=0 is not appropriate for a newly entering coefficient at a positive knot.

Each least angle regression segment starts with all active scores at magnitude $t$. Moving with direction $d_A=G_{AA}^{-1}{s}_A$ changes the score vector to $t{s}_A-(t-\lambda )G_{AA}{d}_A=\lambda s_A$. The first-hit rule bounds every inactive score in magnitude by $\lambda$. This is the key Lasso optimality link.

On a segment starting at residual-score level $t$, an inactive score evolves as $c_j-h{a}_j$ while the active level is $t-h$. Solving $c_j-h{a}_j=\pm (t-h)$ gives distances $(t-c_j)/(1-a_j)$ and $(t+c_j)/(1+a_j)$. The smallest positive admissible distance determines the next entry; reaching level zero ends the algorithm.

A reciprocal-predictor linear regression uses $\mathbb{E}(Y\mid v)=\alpha +\beta /v$ for nonzero predictor $v$. It remains linear in its coefficients. Its flattening shape can suit a saturating response over a positive predictor range; residual checks and cross-validation must assess adequacy, and extrapolation toward zero is hazardous.

The intercept is the coefficient of the constant design column. Adding a constant to a predictor can be absorbed by changing the intercept, leaving fitted values and all slope coefficients unchanged.

For a Gaussian linear model with an intercept and one explanatory variable, $n>2$ and $S_{xx}>0$, the regression intercept estimator is normal with variance ${\sigma }^2(1/n+{\bar{x}}^2/S_{xx})$. The residual variance estimate $s^2=\mathrm{SSE}/(n-2)$ is independent of the coefficient estimators and $(n-2)s^2/{\sigma }^2$ has a chi-squared distribution with $n-2$ degrees of freedom. Consequently the displayed statistic has a Student t-distribution under $\alpha ={\alpha }_0$, and a two-sided level-$\eta$ test rejects for $|T|>t_{n-2,1-\eta /2}$.

// Target: mathematical-optimization.bigb

A linear regression in the coefficients with a polynomial mean function $\mathbb{E}(Y\mid t)={\sum }_{j=0}^d{\beta }_j{t}^j$. Successive degrees define nested models that can be compared by an F-test.

With independent prior distributions $\beta \sim N(\mu ,\Sigma )$ and $s\sim \mathrm{IG}(a,b)$ in a normal distribution regression model, the conditional posterior distribution of $\beta$ has covariance matrix $V=(X^{T}X/s+{\Sigma }^{-1}{)}^{-1}$ and mean $V(X^{T}y/s+{\Sigma }^{-1}\mu )$. The conditional posterior distribution of $s$ is inverse-gamma distribution $\mathrm{IG}(a+n/2,b+\Vert y-X\beta {\Vert }^2/2)$. The shape contains no half-dimension contribution from $\beta$, since its prior distribution is independent of $s$. These conditionals give a two-block Gibbs sampler.

A polynomial regression of degree two is linear in its unknown regression coefficients despite being curved in the predictor. Its design matrix has columns $1,x,x^2$; three distinct predictor values give full column matrix rank.

In an ordinary least squares regression with an intercept and a nonconstant response, $R^2=1-\mathrm{RSS}/{\sum }_i(Y_i-\overline{Y}{)}^2$. It measures the fitted proportion of sample variation relative to an intercept-only baseline. It increases when regressors are added, even if those regressors are unhelpful for prediction. A high value does not establish the assumptions of a normal linear model or rule out a systematic regression residual pattern.

For the five-point example in linear regression, the total centered sum of squares is $10.8$ and the residual sum of squares is $0.8$, so $R^2=25/27\approx 0.926$. This summarizes the fitted reduction in squared error. It does not record the order or shape of the regression residuals, which are worth examining in a residual-versus-fitted plot.

Here is a fully constructed illustration. Use $x=(-2,-1,0,1,2)$ and the same fitted line $3+x$ for both datasets. Add respectively the residual vectorsBoth sum to zero, both have zero inner product with $x$, and both have squared norm $0.56$. Their ordinary least squares fit is therefore exactly the same line, with the same $R^2=10/(10+0.56)=125/132\approx 0.947$. But the first residual is $0.2(x^2-2)$, a systematic quadratic shape, while the second changes direction across the predictor values. These examples show directly which information the scalar $R^2$ discards; neither constructed pattern proves a random-error model.

In a normal linear model with an intercept, $n$ observations and $p$ independent mean coefficients, adjusted $R^2$ isIt compares the residual variance estimate with the unmodelled sample variance, adjusting the raw coefficient of determination for fitted dimension. It can be negative and is not guaranteed to increase when another regressor is added. In penalized regression a related convention uses the fit's effective degrees of freedom instead of $p$; the precise smoothing-software variance convention should be specified.

A residual-versus-fitted plot displays residuals vertically against predicted responses horizontally. Curvature suggests an incorrect conditional-mean model; a fan-shaped spread suggests heteroscedasticity, while isolated large residuals may indicate unusual observations. It does not alone measure leverage or influence: regression leverage records unusual predictor combinations, and Cook's distance combines leverage and residual size.

The figure in coefficient of determination shows why the plot is useful: subtracting the fitted linear trend makes a small curved component easy to see. A visible pattern supplies a reason to investigate the model, while the absence of a pattern in a small sample does not establish that all assumptions hold. Other views address other questions: regression residuals against observation order can reveal temporal dependence, and a normal probability plot examines distributional shape. NIST's statistical handbook develops this use of graphical residual analysis for model validation. ^[ref]

A scale-location plot compares the square root of the absolute standardized regression residual with the fitted values. Systematic changes in level or spread suggest heteroscedasticity. It assesses the error scale, while a residual-versus-fitted plot primarily reveals mean patterns as well as changing spread.

Analysis of variance decomposes variation into contributions associated with model terms and residual error. For nested normal linear models, differences of residual sums of squares give sequential or partial F-tests.

An analysis of variance comparing a quantitative response variable across levels of one factor. The ordinary independent-groups F-test evaluates equality of group population means under the one-way normal linear model. Rejection says that not all means are equal, rather than identifying every differing pair.

A heteroscedastic extension of independent-groups one-way analysis of variance, allowing unequal group variances. It does not make paired or clustered observations independent.

An analysis of variance for a design containing repeated observations on the same experimental units, with an appropriate within-unit error structure. It differs from treating every recorded response as independent. The allocation and covariance assumptions, not just the factor count, determine the applicable tests.

Multivariate analysis of variance compares mean vectors of several groups while accounting for correlations between their response variables. In the one-way normal model, independent observations $Y_{ri}\sim N_p({\mu }_r,\Sigma )$ share a positive-definite covariance matrix. Put $E={\sum }_{r,i}(Y_{ri}-{\bar{Y}}_r)(Y_{ri}-{\bar{Y}}_r{)}^T$ and $H={\sum }_r{n}_r({\bar{Y}}_r-\bar{Y})({\bar{Y}}_r-\bar{Y}{)}^T$. Under equal means, these matrices have independent Wishart distributions with degrees of freedom $N-g$ and $g-1$, respectively. This follows by separating the Gaussian data into orthogonal within-group and between-group projections. The Wilks lambda statistic tests equality of the means using both matrices, rather than independently testing each coordinate.

For groups with sizes $n_r$, means ${\bar{x}}_r$, and overall mean $\bar{x}$, define $W={\sum }_{r,i}(x_{ri}-{\bar{x}}_r)(x_{ri}-{\bar{x}}_r{)}^T$ and $B={\sum }_r{n}_r({\bar{x}}_r-\bar{x})({\bar{x}}_r-\bar{x}{)}^T$. Expanding $x_{ri}-\bar{x}=(x_{ri}-{\bar{x}}_r)+({\bar{x}}_r-\bar{x})$ gives total scatter $T=W+B$ because each within-group residual sum vanishes. Both matrices are positive semidefinite, and $\mathrm{rank}B\le \min (p,g-1)$. They separate within-group noise from between-group mean separation.

For nonsingular within-group scatter $E$ and between-group scatter $H$, the statistic is ${\prod }_j(1+{\theta }_j{)}^{-1}$, where ${\theta }_j$ solve the generalized eigenvalue problem $Hv=\theta Ev$. It is small when the group means differ strongly relative to within-group variation. Maximizing a common-covariance normal likelihood under equal and unrestricted means gives the likelihood ratio ${\Lambda }^{N/2}$. Its determinant factors cancel under any nonsingular change of response coordinates. For one response it is equivalent to the analysis of variance F-test.

An additive two-factor Gaussian model separates row and column effects. With one observation in each of $rc$ cells, constraints fixing one row and column effect leave $r+c-1$ mean parameters and $(r-1)(c-1)$ residual degrees of freedom. Balanced least squares gives fitted values equal to row mean plus column mean minus grand mean. With no cell replication, unrestricted interactions are confounded with residual error; the additive interpretation requires an assumption that those interactions are absent or negligible.

In a fully balanced crossed design, centered indicator columns for the two factors are orthogonal because each pair of levels occurs equally often. Missing cells can change the level-pair counts so that the displayed cross-product becomes nonzero. Then sequential sums of squares depend on term order, although the final additive fit is unchanged. Testing either factor conditional on the other requires its extra sum of squares from the appropriate reduced model.

For observed cells with means $a_i+b_j$, form a bipartite graph joining row $i$ to column $j$ when that cell is observed. If all levels occur, the design rank is $r+c-k$, where $k$ is the number of connected components. Indeed a coefficient perturbation in the kernel satisfies $a_i=-b_j$ on every observed edge, and hence has exactly one freely chosen constant on each component. A connected design has rank $r+c-1$ and is identifiable after a single reference constraint. Missing cells destroy the balanced-design orthogonality of row and column effects, so sequential sums of squares can depend on order.

A two-way analysis of variance with the full interaction and one observation in each observed factor cell has one independently varying fitted cell mean per observation. A cell-indicator design matrix has full row matrix rank, even when some combinations are missing, so the model is a saturated statistical model. Its residual sum of squares is zero and its residual degrees of freedom are zero. Missing combinations make some coefficients nonidentifiable; they do not restore a pure-error estimate. Standard errors and an ordinary residual-based F-test for interactions therefore cannot be recovered merely by fitting this larger model.

For an ordered sequence of nested linear regression models $M_0\subset M_1\subset \cdots$, the sequential sum of squares for the $j$th added term is $\mathrm{RSS}(M_{j-1})-\mathrm{RSS}(M_j)$. These are also called type-I sums of squares. They depend on term order when the design is not orthogonal. Dividing by the number of newly added coefficients and the full model's residual mean square gives the sequential F-test. A covariate added first is tested before adjustment for later terms, whereas a treatment added after the covariate is tested with that covariate already included.

For an orthogonal projection matrix $P$, a sum of squares in ANOVA is $Y^{T}PY=\Vert PY{\Vert }^2$. Mutually orthogonal projections yield an additive decomposition of total squared response magnitude. Removing the grand mean gives the corrected total.

A mean square in ANOVA divides a sum of squares in ANOVA by its positive number $d$ of statistical degrees of freedom. If the corresponding residual subspace has scalar error covariance $\lambda I$, its expectation is $\lambda$. Its scale depends on whether raw observations or group means were projected.

An ANOVA stratum is an orthogonal response subspace representing one level of experimental variation, such as between blocks or within blocks. In a covariance model scalar on each such subspace, its eigenvalue supplies the common variance scale for its projected coordinates. Treatment and residual sums of squares in ANOVA must be compared within appropriate strata.

The within-block ANOVA stratum contains response vectors whose coordinates sum to zero separately in every block. For $b$ blocks of size $k$ its dimension is $b(k-1)$. A shared additive block effect cancels in every vector in this subspace.

In a complete two-factor design with equal cell replication, centred factor indicator columns for different factors have zero cross-products. Consequently the two main effects in an additive normal linear model are orthogonal, and their extra sums of squares do not depend on the order in which these main effects are entered. An omitted interaction term can still require a separate adequacy check.

The Frisch–Waugh–Lovell theorem says that a regression coefficient can be obtained by residualizing both the response and its predictor against the remaining predictors and then regressing one residual on the other.

Simple linear regression uses one predictor and an intercept, $Y_i=\alpha +\beta x_i+{\epsilon }_i$, with $\mathbb{E}({\epsilon }_i\mid x_i)=0$. The regression coefficients describe a straight conditional-mean line: $\alpha$ is its value at zero and $\beta$ its rate of change per unit of the predictor. When zero is outside the observed range, predictor centering can make the intercept refer to a more representative predictor value.

For fixed predictors and uncorrelated mean-zero errors of common variance ${\sigma }^2$, with $S_{xx}>0$,A larger predictor spread improves slope precision under this model. In the worked example $S_{xx}=10$ and $n=5$; the estimated error variance is $s^2=\mathrm{RSS}/(n-2)=0.8/3$. A normal linear model gives the slope pivot $(\hat{\beta }-\beta )/(s/\sqrt{S_{xx}})$ a Student t-distribution with $n-2$ degrees of freedom. The distinction between a fitted line and a reliable uncertainty statement is precisely the role of the error assumptions.

Consider ordinary least squares with an intercept, one nonconstant predictor, $n>2$ cases, $S_{xx}={\sum }_i(x_i-\bar{x}{)}^2>0$ a nonconstant response, and a positive residual sum of squares. With the usual common-variance standard error, put $s^2=\mathrm{RSS}/(n-2)$. ThenThe least-squares normal equations give zero residual sum and zero residual inner product with the centred predictor, proving the first identity and $\hat{\beta }=S_{xy}/S_{xx}$. The fitted centred sum of squares is ${\hat{\beta }}^2{S}_{xx}$. Dividing it by $s^2$ gives the one-predictor F statistic and the squared zero-slope t statistic. Dividing fitted and residual sums of squares by their total gives the last expression. If $0\le R^2<1$, the adjusted coefficient of determination is $1-(1-R^2)(n-1)/(n-2)$.

These are algebraic consistency identities for one specified fit; the mean identity needs no normal-error assumption. A normal linear model supplies the usual finite-sample t and F reference distributions. Heteroscedasticity-robust standard errors need not satisfy the displayed F=t-squared relationship to the classical sums-of-squares F statistic. Rounding can create small discrepancies, while different analysed records or a changed model can create larger ones; neither should be silently treated as the same fit.^[ref]

In simple linear regression written as $Y_i=\alpha +\beta (x_i-\bar{x})+{\epsilon }_i$, the constant and centred predictor columns are orthogonal. Differentiating the sum of squared residuals gives the displayed ordinary least squares estimators, provided $S_{xx}={\sum }_i(x_i-\bar{x}{)}^2>0$. For independent mean-zero errors of common variance ${\sigma }^2$, both estimators are unbiased, their variances are ${\sigma }^2/n$ and ${\sigma }^2/S_{xx}$, and their covariance is zero. The uncentred intercept is $\hat{\alpha }-\hat{\beta }\bar{x}$.

The residual sum of squares measures how far the observed responses lie from their fitted values. If $r_i=Y_i-{\hat{Y}}_i$ is the $i$th regression residual, thenIt is the squared Euclidean norm of the regression residual vector. Squaring prevents positive and negative deviations from cancelling and gives large deviations more weight. In the constructed five-point example in linear regression, the regression residuals are $(1/5,1/5,-4/5,1/5,1/5)$, so the residual sum of squares is $4/5$.

Ordinary least squares minimizes the residual sum of squares over the chosen linear regression family; ridge regression minimizes the residual sum of squares plus a penalty. Consequently the ridge regression fit can have a larger residual sum of squares than the ordinary least squares fit on exactly the same design matrix. For the one-predictor construction in ridge regression, the residual sum of squares rises from $4/5$ to $33/10$ when the penalty changes from $0$ to $10$; that increase is the cost of shrinking the fitted slope.

Dividing the residual sum of squares by $n$ gives the training error for squared-error loss, not an estimate based on fresh responses. Its units are squared response units, and it adds across observations, so raw totals need not be comparable across different response scales or sample sizes. Comparing fits to the same responses is useful, but choosing a ridge regression penalty by minimum training residual sum of squares favors the unpenalized fit. Assess prediction error using a validation set or cross-validation instead.^[ref]

Ridge regression fits a linear regression model while penalizing the size of its slope regression coefficients. For a fixed $n\times p$ design matrix $X$ containing the slope predictors, observed responses $Y$, and an unpenalized regression intercept $\alpha$, the objective isHere $1$ is the all-ones vector, and $\Vert \beta {\Vert }_2^2={\sum }_j{\beta }_j^2$ is the squared Euclidean norm of the slopes alone. The first term is the residual sum of squares; the second implements regularization. No random-error assumption is needed to define this fit. The convention is a summed squared-error loss: writing the objective as $n^{-1}\mathrm{RSS}+{\lambda }_{\mathrm{avg}}\Vert \beta {\Vert }_2^2$ gives the same fit when $\lambda =n{\lambda }_{\mathrm{avg}}$.^[ref]

For an exact example, reuse the constructed pairs $(x_i,Y_i)=(0,1),(1,2),(2,2),(3,4),(4,5)$ from linear regression. Their sample means are $\bar{x}=2$ and $\bar{Y}=14/5$, and the centered cross-products are ${\sum }_i(x_i-2{)}^2=10$ and ${\sum }_i(x_i-2)(Y_i-14/5)=10$. ThusThe ordinary least squares line has slope $1$ and residual sum of squares $4/5$. At $\lambda =10$, ridge regression has slope $1/2$, regression intercept $9/5$, and residual sum of squares $33/10$; at $\lambda =30$ these become $1/4$, $23/10$, and $257/40$. Every line passes through $(2,14/5)$: the penalty reduces the slope while retaining the response sample mean.

The computational benefit is visible after predictor centering. Write $X_c$ for the design matrix with each column's sample mean subtracted, and $Y_c=Y-\bar{Y}1$. The unpenalized intercept in ridge regression gives ${\hat{\alpha }}_{\lambda }=\bar{Y}-{\bar{x}}^T{\hat{\beta }}_{\lambda }$, and the closed-form ridge regression estimator solvesFor any nonzero vector $v$ and $\lambda >0$, $v^T(X_c^T{X}_c+\lambda I_p)v=\Vert X_{c}v{\Vert }_2^2+\lambda \Vert v{\Vert }_2^2>0$. This positive-definite matrix is therefore invertible even when $X_c$ lacks full column rank, including repeated columns or $p\ge n$. The unique penalized slopes resolve the ambiguity of rank-deficient ordinary least squares; they do not supply evidence about slope directions absent from the data. This makes ridge regression useful for multicollinearity.^[ref]

Predictor centering and scaling serve different purposes. Centering handles the free regression intercept. Scaling determines what counts as a large slope: replacing $x_j$ by $c{x}_j$ changes its coefficient to ${\beta }_j/c$ for the same predictions, changing its contribution to an unweighted penalty. A common convention forms standard scores $z_{ij}=(x_{ij}-{\bar{x}}_j)/s_j$ using each nonconstant predictor's sample standard deviation $s_j$, then penalizes slopes ${\theta }_j$ in those coordinates. Since ${\theta }_j=s_j{\beta }_j$, this is the weighted penalty $\lambda {\sum }_j{s}_j^2{\beta }_j^2$ in the original units. Constant columns have $s_j=0$ and must be removed before this scaling. Scaling is a modelling choice; equal penalties in original units and in standardized units define different fits.^[ref]

The singular value decomposition explains the shrinkage. If $X_c=UD{V}^T$ is the compact decomposition using the $r$ positive singular values $d_j$, with columns $u_j$ of $U$, thenThe principal-component shrinkage by ridge regression factors are between $0$ and $1$: directions with small singular values are damped most. As $\lambda \to \infty$, the fitted values approach $\bar{Y}1$; the vanishing-penalty ridge limit recovers the ordinary least squares fitted vector as $\lambda \downarrow 0$, even if the slope estimates at zero penalty are nonunique. The covariance and bias of a ridge regression estimator describes the resulting bias-variance tradeoff. A penalty can reduce variance of an estimator while introducing bias of an estimator; the unbounded directional risk of fixed ridge shrinkage rules out a uniform improvement for a fixed positive penalty over all possible signals.^[ref]

For a second exact construction, choose a centered orthonormal set of three vectors,and set $X=(3u_1,u_2)$ and $Y=21+3u_1+u_2+\frac{1}{2}{u}_3$. The squared singular values are $9$ and $1$. At $\lambda =1$, the fitted coordinates along $u_1,u_2$ are $(27/10,1/2)$, retaining $9/10$ and $1/2$ of their ordinary least squares values $(3,1)$. At $\lambda =9$ they become $(3/2,1/10)$, retaining $1/2$ and $1/10$. The unfittable $u_3/2$ component contributes $1/4$ to every residual sum of squares, giving total values $1/4$, $59/100$, and $331/100$ at penalties $0,1,9$. These are constructed fits, not measured prediction results.

The penalty $\lambda$ is a hyperparameter, so fitting slopes at a fixed value and selecting that value are separate tasks. Use a validation set or K-fold cross-validation to compare candidate penalties by held-out squared-error loss. In each split, compute predictor sample means and scales from that split's training data, apply the same transformation to its validation set, and refit the slopes; using held-out information in preprocessing introduces data leakage. Minimum training residual sum of squares cannot select useful shrinkage: ordinary least squares already attains the smallest possible value.^[ref] After model selection, refit on all training data with the selected penalty. A separate test set or nested cross-validation assesses the entire selection procedure; a low selected validation score alone is no performance guarantee.^[ref]

For likelihood $Y\mid \beta \sim N(X\beta ,{\sigma }^{2}I)$ and independent prior $\beta \sim N(0,{\tau }^{2}I)$, the posterior distribution is Gaussian with covariance $V$ as displayed and mean $(X^{T}X+\lambda I{)}^{-1}{X}^{T}Y$. Its mean and mode therefore coincide with the ridge regression coefficient estimate. A Gaussian prior alone does not imply this conclusion without the stated likelihood.

// Target: foundations-of-mathematics.bigb

For fixed positive ridge regression penalty, choose a unit eigenvector $v$ of $X^{\mathsf{T}}X$ with eigenvalue $d>0$, and signal ${\beta }^0=Bv$. The directional ridge risk is $({\lambda }^2{B}^2+{\sigma }^{2}d)/(d+\lambda {)}^2$, whereas ordinary least squares has risk ${\sigma }^2/d$. Letting $|B|$ grow makes the ridge squared bias arbitrarily large.

For a full-rank centred design, unscaled ridge regression and zero-mean errors with covariance ${\sigma }^{2}I$, put $S=X^{\mathsf{T}}X$ and $A=(S+\lambda I{)}^{-1}$. The ordinary-least-squares risk matrix minus the ridge risk matrix is $\lambda A(2{\sigma }^{2}I+\lambda {\sigma }^2{S}^{-1}-\lambda {\beta }^0{\beta }^{0\mathsf{T}})A$. It is positive definite whenever $\lambda \Vert {\beta }^0{\Vert }_2^2<2{\sigma }^2$, including every positive $\lambda$ when ${\beta }^0=0$. Thus one signal-dependent small penalty improves every nonzero-direction mean squared error.

In ridge regression, penalize slopes but ordinarily leave the regression intercept free. Centering gives $\hat{\beta }=(X_c^T{X}_c+\lambda I{)}^{-1}{X}_c^T{Y}_c$ and $\hat{\alpha }=\overline{Y}-{\overline{x}}^T\hat{\beta }$. An exactly centered predictor matrix has $\hat{\alpha }=\overline{Y}$ for every penalty. A table with a penalty-dependent intercept therefore cannot arise from exactly centered columns under this convention.

For a fixed centered design matrix $X$, independent errors of variance ${\sigma }^2$, and objective $\Vert Y-X\beta {\Vert }^2+a\Vert \beta {\Vert }^2$, put $G=X^{T}X$ and $M=G+aI$. The ridge regression estimator hasThus its standard error alone does not justify centering a confidence interval for $\beta$ at the biased estimator. A penalty chosen from the responses also changes its sampling distribution.

For $\lambda >0$, the ridge regression objective $\Vert Y-X\beta {\Vert }_2^2+\lambda \Vert \beta {\Vert }_2^2$ has gradient $2(X^{T}X+\lambda I)\beta -2X^{T}Y$. The matrix $X^{T}X+\lambda I$ is positive definite, so the unique minimizer is ${\hat{\beta }}_{\lambda }=(X^{T}X+\lambda I{)}^{-1}{X}^{T}Y$.

In an eigenbasis of $G=X^{T}X$, ridge regression divides the corresponding component of $X^{T}Y$ by $\gamma +\lambda$. If $\gamma =0$, its eigenvector $v$ satisfies $Xv=0$, and hence $v^T{X}^{T}Y=0$ exactly. Only positive eigenvalues contribute to the limit, giving the Moore-Penrose inverse expression for the minimum-norm least-squares solution. No full-rank or sample-size assumption is needed.

For any real design matrix $X$ and $\lambda >0$, $(X^{T}X+\lambda I{)}^{-1}{X}^T=X^T(X{X}^T+\lambda I{)}^{-1}$. Multiplication by $X$ identifies the primal ridge regression fitted values with $K(K+\lambda I{)}^{-1}Y$, $K=X{X}^T$, the kernel-ridge hat matrix for the linear kernel. No rank assumption is needed.