= Overdetermined least-squares example
{title2=$x_*=(2,3)^T$}
Suppose three measurements constrain two unknowns:
$$
A=\begin{pmatrix}1&0\\0&1\\1&1\end{pmatrix},
\qquad
b=\begin{pmatrix}1\\2\\6\end{pmatrix}.
$$
The <linear system> asks for $x_1=1$, $x_2=2$, and $x_1+x_2=6$, which are incompatible. A <least-squares solution> makes a balanced compromise. The <matrix> has <full column rank>, so
$$
A^+=(A^TA)^{-1}A^T
=\frac13\begin{pmatrix}2&-1&1\\-1&2&1\end{pmatrix},
\qquad
\boxed{x_*=A^+b=(2,3)^T.}
$$
The fitted output and residual are
$$
Ax_*=(2,3,5)^T,
\qquad
r=b-Ax_*=(-1,-1,1)^T,
\qquad
A^Tr=0.
$$
Thus the two direct measurements are each overshot by one, and the measured sum is undershot by one. The residual has <Euclidean norm> $\sqrt3$ and is <orthogonal> to the <column space>, the <plane> $z=y_1+y_2$ in output coordinates $(y_1,y_2,z)$.
\Image[/pseudoinverse-least-squares-projection.png]
{title=The pseudoinverse projects inconsistent data onto the attainable output plane}
{description=For $b=(1,2,6)^T$, the closest attainable output is $Ax_*=(2,3,5)^T$. The segment between them is the residual $(-1,-1,1)^T$, perpendicular to the <column space>. The drawing is an orthonormal two-dimensional slice of output space; the attainable <plane> intersects this slice in the blue <straight line>.}
For any perturbation $h\in\mathbb R^2$, the <orthogonality> of $r$ to $Ah$ gives
$$
\|b-A(x_*+h)\|_2^2=\|r\|_2^2+\|Ah\|_2^2
=3+\|Ah\|_2^2.
$$
Since the <kernel> of $A$ is trivial, every nonzero $h$ increases the residual. The <least-squares solution> is unique in this example. The formula using $A^TA$ is convenient for these small exact calculations; an actual numerical computation can use a <QR decomposition> or <singular value decomposition> to avoid the extra conditioning cost of the <least-squares normal equations>. For <full column rank>, the squared <singular values> of $A$ are the <eigenvalues> of $A^TA$, so its <condition number> in the <Euclidean norm> is $\kappa_2(A^TA)=\kappa_2(A)^2$. https://ocw.mit.edu/courses/18-065-matrix-methods-in-data-analysis-signal-processing-and-machine-learning-spring-2018/resources/lecture-9-four-ways-to-solve-least-squares-problems/[Strang, MIT 18.065 Lecture 9: four ways to solve least-squares problems]{ref}
Back to article page
