Orthogonal projection onto a finite-dimensional subspace

A finite-dimensional linear subspace $W$ of an inner product space has a unique closest vector to any given vector $x$. This is its orthogonal projection $P_{W}x$. If $e_1,\dots ,e_d$ is an orthogonal basis of $W$, compute it one direction at a time:The inner product convention here is linear in its first argument, including over the complex numbers. For an orthonormal basis, each denominator is one. When $W$ consists only of the zero vector, the empty sum gives $P_{W}x=0$. The finite-dimensional formula is described in Francis Narcowich, Inner Products and Norms: minimization problems.

Take $W=\mathrm{span}\{(1,2{)}^T\}$ and $x=(3,1{)}^T$, using the dot product. With $e=(1,2{)}^T$,The orthogonal projection residual has $r\cdot e=2-2=0$. To verify the best approximation directly, every candidate in the line is $te$, andThe smallest squared Euclidean norm error is therefore $5$, attained only at $t=1$. The diagram connects the perpendicular residual to this one-variable quadratic polynomial.

The general mechanism is the Pythagorean theorem in an inner-product space. For any $w\in W$, the vectors $r=x-P_{W}x$ and $P_{W}x-w$ are orthogonal, soMoving away from $P_{W}x$ inside $W$ adds a nonnegative squared norm, which is zero only when $w=P_{W}x$. This proves both optimality and uniqueness.

The useful feature is separation of preparation from evaluation. Construct an orthonormal basis once with the Gram-Schmidt process, then obtain each new best approximation using inner products instead of a fresh optimization problem. When $W$ is the column space of a matrix, this gives linear least squares; when $W$ is spanned by orthogonal polynomials, it gives a least-squares polynomial in an orthogonal-polynomial basis. The ambient inner product space need not be a Hilbert space for this finite-dimensional construction. In an infinite-dimensional target linear subspace, existence requires additional hypotheses, supplied by orthogonal decomposition by a closed subspace.

In the weighted inner product $⟨f,g⟩={\int }_0^{1}xf(x)g(x)dx$, the best approximation to $x^2$ in the span of $1,x$ is the displayed polynomial. The normal equations are $1/4-A/2-B/3=0$ and $1/5-A/3-B/4=0$. Their solution gives $A=-3/10$, $B=6/5$. The residual is orthogonal to that span, so the Pythagorean theorem proves minimality and uniqueness. Its squared norm is $1/600$. A weight vanishing at a single endpoint still gives a positive-definite inner product on continuous functions: a nonzero continuous function remains nonzero on an interval where the weight is positive.

An orthogonal projection residual is the difference $r=x-P_{W}x$ between a vector and its orthogonal projection onto a linear subspace $W$. It is orthogonal to every vector in $W$, so $⟨r,w⟩=0$ for all $w\in W$. The Gram-Schmidt process uses such residuals to remove directions already represented by earlier basis vectors.

This is an optimality condition as well as a geometric description. No permitted change within $W$ can remove a further orthogonal component of $r$. In the preceding dot product example, $r=(2,-1{)}^T$ is perpendicular to the line spanned by $(1,2{)}^T$.

The selected inner product matters. Keep $x=(3,1{)}^T$ and the same line, but use the weighted Euclidean inner product with $M=\mathrm{diag}(4,1)$. The projection coefficient and residual becomeNow $4(r_M{)}_1{e}_1+(r_M{)}_2{e}_2=5-5=0$, although $r_M\cdot e=-15/4\ne 0$. The orthogonality is with respect to $M$, so it need not look perpendicular in the ordinary drawing. The squared weighted error isThe weighted optimum reduces the first-coordinate error from $2$ to $5/4$, at the cost of increasing the magnitude of the second-coordinate error from $1$ to $5/2$. The weights express which errors are more costly.

In linear least squares, residual orthogonality to the column space is expressed by the normal equations for linear least squares. With a weighted Euclidean inner product, the corresponding real-matrix equation is $A^{T}M(x-Ac)=0$. This makes the choice of error measurement part of the mathematical model, rather than merely a plotting convention.

A matrix represents a linear projection when applying it twice has the same effect as applying it once. For an orthogonal projection with the standard Hermitian inner product on ${\mathbb{C}}^m$, the additional condition is that the conjugate transpose equals the original matrix:Thus $P$ is both idempotent and a Hermitian matrix. Over the real numbers this says $P^2=P=P^T$. Its image of a linear map is the retained linear subspace, while its kernel of a linear map is the orthogonal complement: vectors in the image are unchanged, and vectors in the kernel become zero.

For a matrix $A$ with full column rank, projecting onto its column space givesThe invertibility of $A^{†}A$ follows from linear independence of the columns. If those columns are an orthonormal set, $A^{†}A=I$ and $P=A{A}^{†}$. These formulas and the conditions $P^2=P=P^T$ in the real case are developed in MIT 18.06SC, Projections onto Subspaces, pp. 1–3.

For the line spanned by $e=(1,2{)}^T$,In particular, $Pe=e$ and $P(2,-1{)}^T=0$. The eigenvalues are $1$ on the retained direction and $0$ on the removed direction. This exposes the meaning of an orthogonal projection as retaining selected components, rather than stretching them.

Symmetry refers to the specified inner product. In the real weighted setting, the formula becomes $P_M=A(A^{T}MA{)}^{-1}{A}^{T}M$ and satisfies $P_M^2=P_M$ and $P_M^{T}M=M{P}_M$. For the preceding weighted line example,It is not symmetric under the ordinary transpose, but it is an orthogonal projection for the weighted Euclidean inner product. The matrix criterion $P=P^{†}$ should therefore be used with the standard Hermitian inner product, not silently transferred to a different geometry.

Under orthogonal projection from one plane onto another, a circle of radius $R$ becomes an ellipse with semiaxes $R$ and $R|\cos \alpha |$, where $\alpha$ is the angle between the planes' unit normal vectors. To prove this, choose one in-plane unit vector parallel to the planes' intersection and the other perpendicular to it. The first vector is unchanged by the projection; the second has projected length $|\cos \alpha |$ and remains perpendicular to the first. Parametrizing the circle in that orthonormal basis therefore gives projected coordinates $(R\cos t,R\cos \alpha \sin t)$. Parallel planes have a circular image, and perpendicular planes give the degenerate case of a line segment. An arbitrary center simply projects to the center of the image.

For a nonnull vector $k$ in a space with a nondegenerate bilinear form, $P_{\mu \nu }={\eta }_{\mu \nu }-k_{\mu }{k}_{\nu }/k^2$ projects onto the subspace transverse to $k$. It obeys $P_{\mu \nu }{k}^{\nu }=0$ and $P^2=P$.

For orthogonal polynomials $Q_0,\dots ,Q_n$ under a positive weighted integral inner product, the least-squares approximation to $f$ of degree at most $n$ isIts residual is orthogonal to every polynomial of degree at most $n$.