 Orthogonal projection

The orthogonal projection $P_F$ onto a closed subspace of a Hilbert space $F$ sends $x$ to its component in the orthogonal decompositionThe residual $r$ is orthogonal to every permitted direction. For any $z\in F$, the Pythagorean theorem givesThus $P_{F}x$ uniquely minimizes the distance to $F$. It is a linear map, is idempotent, and is nonexpansive: $\Vert P_{F}x-P_{F}y\Vert \le \Vert x-y\Vert$; taking $y=0$ gives $\Vert P_{F}x\Vert \le \Vert x\Vert$.

In ${\mathbb{R}}^2$, project $v=(1,3)$ onto the line spanned by $u=(2,1)$. The inner product givesThe check $⟨r,u⟩=-2+2=0$ confirms orthogonality, and $\Vert v{\Vert }^2=10=5+5=\Vert P_{F}v{\Vert }^2+\Vert r{\Vert }^2$. Projecting again changes nothing, since the result already lies on the line.

For a finite orthonormal basis $e_1,\dots ,e_m$ of $F$, the same operation is $P_{F}x={\sum }_j⟨x,e_j⟩e_j$, using the convention linear in the first argument. Replacing coordinate vectors by functions gives exactly the same approximation principle. For $f(x)=x^2$ in the real L2 space on $[-1,1]$ and $F=\mathrm{span}\{1,x\}$, the directions are orthogonal, with squared norms $2$ and $2/3$. Their inner products with $f$ are $2/3$ and zero, soDirect integration gives the squared error $\Vert x^2-1/3{\Vert }^2=8/45$. For any competing affine polynomial, the orthogonal residual yieldsEvery deviation from $a=1/3$, $b=0$ adds a nonnegative error. The best function need not pass through any selected data point: it minimizes an integrated error across the whole interval.