= Orthogonal projection onto a finite-dimensional subspace
A finite-dimensional <linear subspace> $W$ of an <inner product space> has a unique closest <vector> to any given <vector> $x$. This is its <orthogonal projection> $P_Wx$. If $e_1,\ldots,e_d$ is an <orthogonal basis> of $W$, compute it one direction at a time:
$$
P_Wx=\sum_{j=1}^d\frac{\langle x,e_j\rangle}{\langle e_j,e_j\rangle}e_j.
$$
The <inner product> convention here is linear in its first argument, including over the <complex numbers>. For an <orthonormal basis>, each denominator is one. When $W$ consists only of the <zero vector>, the empty sum gives $P_Wx=0$. The finite-dimensional formula is described in https://people.tamu.edu/~f-narcowich/m641/m641_notes/inner_products.html[Francis Narcowich, Inner Products and Norms: minimization problems].
Take $W=\operatorname{span}\{(1,2)^T\}$ and $x=(3,1)^T$, using the <dot product>. With $e=(1,2)^T$,
$$
\frac{x\cdot e}{e\cdot e}=\frac{3+2}{1+4}=1,\qquad
P_Wx=(1,2)^T,\qquad r=x-P_Wx=(2,-1)^T.
$$
The <orthogonal projection residual> has $r\cdot e=2-2=0$. To verify the best approximation directly, every candidate in the line is $te$, and
$$
\|x-te\|_2^2=(3-t)^2+(1-2t)^2=5(t-1)^2+5.
$$
The smallest squared <Euclidean norm> error is therefore $5$, attained only at $t=1$. The diagram connects the perpendicular residual to this one-variable <quadratic polynomial>.
\Image[/orthogonal-projection-closest-point.png]
{title=The perpendicular residual identifies the closest point on a line and the unique minimum of the squared error}
The general mechanism is the <Pythagorean theorem in an inner-product space>. For any $w\in W$, the <vectors> $r=x-P_Wx$ and $P_Wx-w$ are <orthogonal>, so
$$
\|x-w\|^2=\|x-P_Wx\|^2+\|P_Wx-w\|^2.
$$
Moving away from $P_Wx$ inside $W$ adds a nonnegative squared <norm>, which is zero only when $w=P_Wx$. This proves both optimality and uniqueness.
The useful feature is separation of preparation from evaluation. Construct an <orthonormal basis> once with the <Gram-Schmidt process>, then obtain each new best approximation using <inner products> instead of a fresh <optimization> problem. When $W$ is the <column space> of a <matrix>, this gives <linear least squares>; when $W$ is spanned by <orthogonal polynomials>, it gives a <least-squares polynomial in an orthogonal-polynomial basis>. The ambient <inner product space> need not be a <Hilbert space> for this finite-dimensional construction. In an infinite-dimensional target <linear subspace>, existence requires additional hypotheses, supplied by <orthogonal decomposition by a closed subspace>.
Back to article page
