= Linear regression
{title2=$Y=X\beta+\varepsilon$}
{wiki}
Linear regression models a response's <conditional expectation> as a <linear combination> of specified predictors. In <matrix> form,
$$
Y=X\beta+\varepsilon,\qquad \mathbb E(\varepsilon\mid X)=0,
$$
where the <design matrix> $X$ describes the observations, $\beta$ contains the <regression coefficients>, and $\varepsilon$ contains the errors. The <fitted values> $X\widehat\beta$ estimate the response means. The model is linear in its <regression coefficients>; its columns may include powers, interactions, or other transformed predictors, so <polynomial regression> can fit curved relationships using the same machinery.
For a concrete example, take five constructed observations
$$
(x_i,y_i)=(0,1),(1,2),(2,2),(3,4),(4,5).
$$
An <ordinary least squares> fit with an <intercept> gives the line $\widehat y=0.8+x$. Its <regression residuals> are $(0.2,0.2,-0.8,0.2,0.2)$ and its <residual sum of squares> is $0.8$. A one-unit increase in the predictor corresponds to a one-unit increase in the fitted response. This is a description of this fitted association; interpreting a change as a <causal effect> requires the assumptions of a <causal inference> problem.
\Image[/linear-regression-worked-example.png]
{title=A five-point least-squares fit separates the fitted trend from residual deviations}
The geometric reason for the fit is simple. The possible mean vectors $X\beta$ form the <column space> of $X$. <Ordinary least squares> chooses the <orthogonal projection> of the data vector onto that <vector subspace>, so the remaining <regression residual> is <orthogonal> to every permitted fitted direction. The <least-squares normal equations> express this orthogonality as $X^T(Y-X\widehat\beta)=0$. The data's explained component and unexplained component are therefore separated by the same geometry used in a <Hilbert space>.
The optimization definition makes sense for any observed data. Quantifying the uncertainty of the <regression coefficients> needs additional assumptions about the sampling and errors. With fixed $X$ of <full column rank> and mean-zero errors with <covariance matrix> $\sigma^2I$, <ordinary least squares> is unbiased and has covariance $\sigma^2(X^TX)^{-1}$; the <Gauss-Markov theorem> compares it with other linear unbiased estimators. A <normal linear model> additionally makes <Student t-distribution> intervals and tests exact in finite samples. Model adequacy and predictive assessment remain relevant even when these calculations are available.
Two historical settings help explain the subject's appeal. Adrien-Marie Legendre's \i[Nouvelles méthodes pour la détermination des orbites des comètes] (1805) includes an appendix on the method of least squares: observations give more equations than unknowns, and squared errors provide a criterion for reconciling them. Its astronomical setting shows the motivation for combining imperfect measurements. https://www.e-rara.ch/zut/content/titleinfo/10838079[Legendre, Nouvelles méthodes pour la détermination des orbites des comètes, 1805, appendix pp. 72–80]{ref} Francis Galton's \i[Regression Towards Mediocrity in Hereditary Stature] (1886) examines the association between parental and offspring heights and the tendency of group averages toward the population average. This is a historical use of the regression name; the modern linear model is a general statistical formulation. https://zenodo.org/records/1449548[Galton, Regression Towards Mediocrity in Hereditary Stature, 1886, original article]{ref}
Back to article page
