= Normal equations for linear least squares
{title2=$X^TX\widehat\theta=X^Ty$}
= Least-squares normal equations
{synonym}
Minimizing $\|y-X\theta\|^2$ gives the displayed equations by <differentiation>. Equivalently, the <regression residual> $r=y-X\widehat\theta$ satisfies $X^Tr=0$: it is <orthogonal> to every column of the <design matrix>. If $X$ has <full column rank>, the minimizer is unique and the <fitted values> are the <orthogonal projection> of $y$ onto its <column space>. A rank-deficient <design matrix> can give multiple coefficient vectors, while the projected fitted vector is still unique.
For an <intercept> and one predictor, put $S_{xx}=\sum_i(x_i-\bar x)^2$ and $S_{xy}=\sum_i(x_i-\bar x)(y_i-\bar y)$. The equations give $\widehat\beta=S_{xy}/S_{xx}$ and $\widehat\alpha=\bar y-\widehat\beta\bar x$, provided $S_{xx}>0$. In the five-point example in <linear regression>, $\bar x=2$, $\bar y=2.8$, $S_{xx}=10$ and $S_{xy}=10$, giving $\widehat\beta=1$ and $\widehat\alpha=0.8$.
The <normal equations> describe the mathematical optimum, but explicitly forming and inverting $X^TX$ can amplify numerical problems. A <QR decomposition> computes the same <least-squares solution> through orthogonal coordinates; a <singular value decomposition> also exposes nearly dependent columns. This connects statistical fitting to <numerical linear algebra> and the choice of a stable algorithm. https://ocw.mit.edu/courses/18-06sc-linear-algebra-fall-2011/[MIT 18.06SC, Projections onto Subspaces and Least Squares]{ref}
Back to article page
