= Orthogonal projection residual
{title2=$r=x-P_Wx$}
An <orthogonal projection residual> is the difference $r=x-P_Wx$ between a <vector> and its <orthogonal projection> onto a <linear subspace> $W$. It is <orthogonal> to every <vector> in $W$, so $\langle r,w\rangle=0$ for all $w\in W$. The <Gram-Schmidt process> uses such residuals to remove directions already represented by earlier basis <vectors>.
This is an optimality condition as well as a geometric description. No permitted change within $W$ can remove a further <orthogonal> component of $r$. In the preceding <dot product> example, $r=(2,-1)^T$ is perpendicular to the line spanned by $(1,2)^T$.
The selected <inner product> matters. Keep $x=(3,1)^T$ and the same line, but use the <weighted Euclidean inner product> with $M=\operatorname{diag}(4,1)$. The projection coefficient and residual become
$$
t_M=\frac{\langle x,e\rangle_M}{\langle e,e\rangle_M}
=\frac{14}{8}=\frac74,\qquad
P_W^{(M)}x=\left(\frac74,\frac72\right)^T,\qquad
r_M=\left(\frac54,-\frac52\right)^T.
$$
Now $4(r_M)_1e_1+(r_M)_2e_2=5-5=0$, although $r_M\cdot e=-15/4\ne0$. The <orthogonality> is with respect to $M$, so it need not look perpendicular in the ordinary drawing. The squared weighted error is
$$
\|x-te\|_M^2=4(3-t)^2+(1-2t)^2
=8\left(t-\frac74\right)^2+\frac{25}{2}.
$$
The weighted optimum reduces the first-coordinate error from $2$ to $5/4$, at the cost of increasing the magnitude of the second-coordinate error from $1$ to $5/2$. The weights express which errors are more costly.
\Image[/orthogonal-projection-metric-comparison.png]
{title=Changing the inner product changes the closest point: a circle and a weighted error ellipse touch the same permitted line at different points}
In <linear least squares>, residual <orthogonality> to the <column space> is expressed by the <normal equations for linear least squares>. With a <weighted Euclidean inner product>, the corresponding real-matrix equation is $A^TM(x-Ac)=0$. This makes the choice of error measurement part of the mathematical model, rather than merely a plotting convention.
Back to article page
