Linear regression

Linear regression models a response mean as a linear combination $X\beta$ of predictors. Ordinary least squares chooses $\beta$ to minimize the sum of squared residuals.

If the true conditional mean is ${\beta }_0+{\beta }_{1}X+{\beta }_{2}Z$, the population slope from regressing on $X$ alone is ${\beta }_1+{\beta }_2\mathrm{Cov}(X,Z)/\mathrm{Var}(X)$, assuming a mean-zero error uncorrelated with the predictors. The omitted predictor can steepen, flatten, or reverse the pooled association. The same identity holds for fitted coefficients and empirical covariances when comparing nested ordinary least-squares fits. Adjustment changes the comparison being described; a causal interpretation requires additional assumptions.

Replacing a numerical predictor $x$ by $x-c$ preserves the fitted mean space and slope in a linear regression containing an intercept. The intercept becomes the fitted mean at $x=c$. In simple linear regression, centering at the sample mean makes intercept and slope estimates uncorrelated under a normal linear model. This changes coefficient interpretation, not predictions.

Regression diagnostics check assumptions and influential observations in a fitted regression function. Residual-versus-fitted plots can reveal omitted mean structure or nonconstant variance; a quantile-quantile plot checks a specified error distribution. Regression leverage measures unusual predictor configurations, and Cook's distance combines leverage and residual size to measure coefficient sensitivity. Independence may require checking collection order and the study design in addition to residual plots.

An influential observation substantially changes a fitted model or its predictions when removed or perturbed. In a linear regression, influence depends jointly on residual size and regression leverage. Large predictor leverage with a small residual and a large residual at ordinary leverage can have different consequences. Regression diagnostics examine this sensitivity; influence is not synonymous with an invalid observation.

A regression outlier has a response unusually far from its fitted conditional mean. A standardized regression residual adjusts for regression leverage; an externally studentized residual also estimates noise variance with the observation deleted. Selecting the largest residual among many observations changes its reference distribution. An outlier need not be an influential observation, and unusual valid data should not be discarded solely for being unusual.

A piecewise linear regression-path algorithm which starts at zero, activates a predictor with maximal absolute residual score, and moves in a direction that decreases all active absolute scores equally until another predictor ties them. With $G=X^{\mathsf{T}}X/n$ and active correlation signs $s_A$, the coefficient direction is $G_{AA}^{-1}{s}_A$. Ordinary LAR does not drop a predictor when its coefficient crosses zero; sign compatibility of LAR and Lasso characterizes when its path also obeys the Lasso KKT conditions.

If every nonzero active coefficient has the sign of its residual score along the least angle regression path, the LAR active correlation invariant and inactive bounds give the Karush-Kuhn-Tucker conditions for the Lasso. A unique Lasso solution then equals the LAR path. Zero coefficients use a subgradient interval; literal sign equality with sign(0)=0 is not appropriate for a newly entering coefficient at a positive knot.

Each least angle regression segment starts with all active scores at magnitude $t$. Moving with direction $d_A=G_{AA}^{-1}{s}_A$ changes the score vector to $t{s}_A-(t-\lambda )G_{AA}{d}_A=\lambda s_A$. The first-hit rule bounds every inactive score in magnitude by $\lambda$. This is the key Lasso optimality link.

On a segment starting at residual-score level $t$, an inactive score evolves as $c_j-h{a}_j$ while the active level is $t-h$. Solving $c_j-h{a}_j=\pm (t-h)$ gives distances $(t-c_j)/(1-a_j)$ and $(t+c_j)/(1+a_j)$. The smallest positive admissible distance determines the next entry; reaching level zero ends the algorithm.

A reciprocal-predictor linear regression uses $\mathbb{E}(Y\mid v)=\alpha +\beta /v$ for nonzero predictor $v$. It remains linear in its coefficients. Its flattening shape can suit a saturating response over a positive predictor range; residual checks and cross-validation must assess adequacy, and extrapolation toward zero is hazardous.

The intercept is the coefficient of the constant design column. Adding a constant to a predictor can be absorbed by changing the intercept, leaving fitted values and all slope coefficients unchanged.

A linear regression in the coefficients with a polynomial mean function $\mathbb{E}(Y\mid t)={\sum }_{j=0}^d{\beta }_j{t}^j$. Successive degrees define nested models that can be compared by an F-test.

A polynomial regression of degree two is linear in its unknown regression coefficients despite being curved in the predictor. Its design matrix has columns $1,x,x^2$; three distinct predictor values give full column matrix rank.

In an ordinary least squares regression with an intercept and a nonconstant response, $R^2=1-\mathrm{RSS}/{\sum }_i(Y_i-\overline{Y}{)}^2$. It measures the fitted proportion of sample variation relative to an intercept-only baseline. It increases when regressors are added, even if those regressors are unhelpful for prediction. A high value does not establish the assumptions of a normal linear model or rule out a systematic regression residual pattern.

A residual-versus-fitted plot displays residuals vertically against predicted responses horizontally. Curvature suggests an incorrect conditional-mean model; a fan-shaped spread suggests heteroscedasticity, while isolated large residuals may indicate unusual observations. It does not alone measure leverage or influence: regression leverage records unusual predictor combinations, and Cook's distance combines leverage and residual size.

A scale-location plot compares the square root of the absolute standardized regression residual with the fitted values. Systematic changes in level or spread suggest heteroscedasticity. It assesses the error scale, while a residual-versus-fitted plot primarily reveals mean patterns as well as changing spread.

Analysis of variance decomposes variation into contributions associated with model terms and residual error. For nested normal linear models, differences of residual sums of squares give sequential or partial F-tests.

For an orthogonal projection matrix $P$, a sum of squares in ANOVA is $Y^{T}PY=\Vert PY{\Vert }^2$. Mutually orthogonal projections yield an additive decomposition of total squared response magnitude. Removing the grand mean gives the corrected total.

A mean square in ANOVA divides a sum of squares in ANOVA by its positive number $d$ of statistical degrees of freedom. If the corresponding residual subspace has scalar error covariance $\lambda I$, its expectation is $\lambda$. Its scale depends on whether raw observations or group means were projected.

An ANOVA stratum is an orthogonal response subspace representing one level of experimental variation, such as between blocks or within blocks. In a covariance model scalar on each such subspace, its eigenvalue supplies the common variance scale for its projected coordinates. Treatment and residual sums of squares in ANOVA must be compared within appropriate strata.

The within-block ANOVA stratum contains response vectors whose coordinates sum to zero separately in every block. For $b$ blocks of size $k$ its dimension is $b(k-1)$. A shared additive block effect cancels in every vector in this subspace.

In a complete two-factor design with equal cell replication, centred factor indicator columns for different factors have zero cross-products. Consequently the two main effects in an additive normal linear model are orthogonal, and their extra sums of squares do not depend on the order in which these main effects are entered. An omitted interaction term can still require a separate adequacy check.

The Frisch–Waugh–Lovell theorem says that a regression coefficient can be obtained by residualizing both the response and its predictor against the remaining predictors and then regressing one residual on the other.

Simple linear regression uses one predictor and an intercept, $Y_i=\alpha +\beta x_i+{\epsilon }_i$.

The residual sum of squares is

Ridge regression minimizes a squared-error loss plus the quadratic penalty $\lambda \Vert \beta {\Vert }_2^2$. Its positive penalty stabilizes inversion in collinear or high-dimensional designs.

For fixed positive ridge regression penalty, choose a unit eigenvector $v$ of $X^{\mathsf{T}}X$ with eigenvalue $d>0$, and signal ${\beta }^0=Bv$. The directional ridge risk is $({\lambda }^2{B}^2+{\sigma }^{2}d)/(d+\lambda {)}^2$, whereas ordinary least squares has risk ${\sigma }^2/d$. Letting $|B|$ grow makes the ridge squared bias arbitrarily large.

For a full-rank centred design, unscaled ridge regression and zero-mean errors with covariance ${\sigma }^{2}I$, put $S=X^{\mathsf{T}}X$ and $A=(S+\lambda I{)}^{-1}$. The ordinary-least-squares risk matrix minus the ridge risk matrix is $\lambda A(2{\sigma }^{2}I+\lambda {\sigma }^2{S}^{-1}-\lambda {\beta }^0{\beta }^{0\mathsf{T}})A$. It is positive definite whenever $\lambda \Vert {\beta }^0{\Vert }_2^2<2{\sigma }^2$, including every positive $\lambda$ when ${\beta }^0=0$. Thus one signal-dependent small penalty improves every nonzero-direction mean squared error.

In ridge regression, penalize slopes but ordinarily leave the regression intercept free. Centering gives $\hat{\beta }=(X_c^T{X}_c+\lambda I{)}^{-1}{X}_c^T{Y}_c$ and $\hat{\alpha }=\overline{Y}-{\overline{x}}^T\hat{\beta }$. An exactly centered predictor matrix has $\hat{\alpha }=\overline{Y}$ for every penalty. A table with a penalty-dependent intercept therefore cannot arise from exactly centered columns under this convention.

For a fixed centered design matrix $X$, independent errors of variance ${\sigma }^2$, and objective $\Vert Y-X\beta {\Vert }^2+a\Vert \beta {\Vert }^2$, put $G=X^{T}X$ and $M=G+aI$. The ridge regression estimator hasThus its standard error alone does not justify centering a confidence interval for $\beta$ at the biased estimator. A penalty chosen from the responses also changes its sampling distribution.

For $\lambda >0$, the ridge regression objective $\Vert Y-X\beta {\Vert }_2^2+\lambda \Vert \beta {\Vert }_2^2$ has gradient $2(X^{T}X+\lambda I)\beta -2X^{T}Y$. The matrix $X^{T}X+\lambda I$ is positive definite, so the unique minimizer is ${\hat{\beta }}_{\lambda }=(X^{T}X+\lambda I{)}^{-1}{X}^{T}Y$.

In an eigenbasis of $G=X^{T}X$, ridge regression divides the corresponding component of $X^{T}Y$ by $\gamma +\lambda$. If $\gamma =0$, its eigenvector $v$ satisfies $Xv=0$, and hence $v^T{X}^{T}Y=0$ exactly. Only positive eigenvalues contribute to the limit, giving the Moore-Penrose inverse expression for the minimum-norm least-squares solution. No full-rank or sample-size assumption is needed.

For any real design matrix $X$ and $\lambda >0$, $(X^{T}X+\lambda I{)}^{-1}{X}^T=X^T(X{X}^T+\lambda I{)}^{-1}$. Multiplication by $X$ identifies the primal ridge regression fitted values with $K(K+\lambda I{)}^{-1}Y$, $K=X{X}^T$, the kernel-ridge hat matrix for the linear kernel. No rank assumption is needed.

For $X^{T}X=V\Lambda V^T$, ridge regression multiplies the fitted response along the $i$th normalized sample principal component by the shrinkage factor ${\Lambda }_{ii}/({\Lambda }_{ii}+\lambda )$. Directions with variance much larger than $\lambda$ are nearly retained, while directions with variance much smaller than $\lambda$ are nearly removed.

Fitted values are the responses predicted at the training covariates by a fitted regression model.

A linear smoother has fitted-value vector $\hat{Y}=HY$, where the smoother matrix $H$ depends on the covariates and tuning parameters but not on the response vector.

For a linear smoother $\hat{Y}=HY$, its effective degrees of freedom are $\mathrm{tr}(H)$. This measures sensitivity of fitted values to responses even when $H$ is not an orthogonal projection.

The design matrix has one row for each observation and one column for each regression coefficient, so the linear predictor is $X\beta$.

If each level of one categorical variable belongs to exactly one level of another, indicators for the coarser levels are sums of indicators for the finer levels. Including unrestricted fixed effects for both therefore gives a rank-deficient ordinary least squares: the separate effects lack identifiability without additional constraints.

A regression coefficient is a component of $\beta$ measuring the change in the linear predictor associated with its design-matrix column.

If $Y=\beta Z+\epsilon$ but the predictor is observed as $X=Z+\eta$, with independent centred errors, the population regression slope isMeasurement error in the predictor therefore shrinks the slope toward zero, while independent response noise changes its uncertainty but not its population value.