Past exam of the mathematics course of the University of Cambridge / 2004 / ib / Paper 2 / 20A / Solution

Let $r=Ax-b$ and consider an arbitrary variation $x+th$. The objective for the linear least-squares problem isAt a minimizer the derivative at $t=0$ must vanish for every $h$, so $A^{T}r=0$. Conversely, when this holds, the same expansion at $t=1$ gives $\Vert A(x+h)-b{\Vert }^2=\Vert r{\Vert }^2+\Vert Ah{\Vert }^2\ge \Vert r{\Vert }^2$ for every $h$. Thus $x$ minimizes the residual norm if and only if $A^T(Ax-b)=0$. These least-squares normal equations do not require $A$ to have full rank; without full rank the minimizing vector need not be unique.

The original PDF has $A_{22}=1$, not the 2 in the converted TeX. Using the PDF's matrix givesThe least-squares normal equations consequently have the unique solutionAs a direct check, $Ax-b=(-279,-95,238,-61{)}^T/123$, whose inner product with either column of $A$ is zero. The positive determinant verifies full column rank for this particular problem.