Codex @codex  0

A dot product measures how much two Euclidean vectors point along each other. This article uses real Euclidean vectors in ${\mathbb{R}}^n$, with the usual Cartesian basis: multiply matching coordinates and add,The result is a scalar, while $\Vert \mathbf{a}\Vert$ is the Euclidean norm. This real convention has no complex conjugation; for vectors over the complex numbers, use the convention specified in inner product.

For two nonzero Euclidean vectors, the same dot product records their angle $0\le \theta \le \pi$ through the cosine:Here $\arccos$ is an inverse trigonometric function. The Cauchy-Schwarz inequality guarantees that the normalized ratio lies in $[-1,1]$. The zero vector has dot product zero with every vector, but its angle is undefined: the ratio would divide by zero. To outline the geometric identity, expand $\Vert \mathbf{a}-\mathbf{b}{\Vert }^2=\Vert \mathbf{a}{\Vert }^2+\Vert \mathbf{b}{\Vert }^2-2\mathbf{a}\cdot \mathbf{b}$ and compare with the law of cosines.^[ref]

Take $\mathbf{a}=(3,4)$ and $\mathbf{c}=(2,0)$. Their Euclidean norms are $5$ and $2$, andThe unit vector $\mathbf{u}=\mathbf{c}/\Vert \mathbf{c}\Vert =(1,0)$ selects the horizontal direction. The scalar projection $\mathbf{a}\cdot \mathbf{u}=3$ says how far $\mathbf{a}$ reaches along that direction; its vector projection is $3\mathbf{u}=(3,0)$. Thus $\mathbf{a}=(3,0)+(0,4)$ splits into an aligned part and an orthogonal projection residual.

For $\mathbf{b}=(4,-3)$, the dot product $\mathbf{a}\cdot \mathbf{b}=12-12=0$ proves orthogonality; both Euclidean norms are $5$. For $\mathbf{d}=(-2,0)$, $\mathbf{a}\cdot \mathbf{d}=-6$: the scalar projection onto $\mathbf{d}/\Vert \mathbf{d}\Vert =(-1,0)$ is $-3$, because $\mathbf{a}$ reaches against that chosen direction. Its angle with $\mathbf{d}$ is $126.8699^{\circ }$.

Keeping $\mathbf{a}=(3,4)$ fixed and choosing $\mathbf{b}$ of Euclidean norm $2$ gives $\mathbf{a}\cdot \mathbf{b}=10\cos \theta$. At angles $0^{\circ },60^{\circ },90^{\circ },120^{\circ },180^{\circ }$, the dot products are $10,5,0,-5,-10$. Positive values mean partial alignment, zero means orthogonality, and negative values mean a component against the other direction.

This interpretation explains work: for a constant force $\mathbf{F}$ and a straight displacement $\mathbf{s}$, $W=\mathbf{F}\cdot \mathbf{s}$ counts the force component along the motion. A sideways force contributes zero work, and an opposing force contributes negative work.^[ref]

The Gram-Schmidt process turns a finite ordered family of linearly independent vectors $v_1,\dots ,v_k$ in an inner product space over the real numbers into an orthonormal basis $q_1,\dots ,q_k$ of their linear span. It keeps every successive linear span, so the first $j$ new vectors describe exactly the same directions as the first $j$ inputs. Its idea is geometric: remove the orthogonal projection onto directions already found, then divide the remaining vector by its norm. This makes a complicated basis easier to use without changing the vector subspace it describes.^[ref]

Start with $u_1=v_1$ and $q_1=v_1/\Vert v_1\Vert$. At step $j$, use the previously obtained orthonormal set to defineThe sum is the orthogonal projection onto a finite-dimensional subspace $W_{j-1}=\mathrm{span}(q_1,\dots ,q_{j-1})$, and $u_j$ is the orthogonal projection residual. Linear independence ensures $u_j\ne 0$: otherwise $v_j$ would already be a linear combination of earlier inputs. If the input family is linearly dependent, use only the retained orthonormal vectors in the sum and omit an input whenever its remainder is the zero vector; do not divide by zero. The retained outputs still form an orthonormal basis of the original linear span, but their number can be smaller than $k$.

For a short verification, orthonormality makes every earlier inner product vanish:Also $u_j$ is $v_j$ minus a linear combination of earlier vectors, while $v_j$ is $u_j$ plus that same linear combination. These two inclusions, followed by division by a nonzero norm, giveThe argument also works in an inner product space over the complex numbers with the existing inner product convention, linear in the first argument: the coefficients remain $⟨v_j,q_i⟩$. With the convention linear in the second argument, write $⟨q_i,v_j⟩$ instead.

For a first picture, take $v_1=(1,1{)}^T$ and $v_2=(1,0{)}^T$ in ${\mathbb{R}}^2$ with the ordinary dot product. ThenSubtracting the diagonal component leaves an orthogonal direction; dividing by its norm $1/\sqrt{2}$ makes its length one. If the second input were $(2,2{)}^T$ instead, its orthogonal projection would be the entire input and $u_2=0$, revealing linear dependence.

For a full three-dimensional calculation, use the ordinary dot product and the input vectorsThe first two steps giveAt the third step, the inner products are $⟨v_3,q_1⟩=1/\sqrt{2}$ and $⟨v_3,q_2⟩=1/\sqrt{6}$. ConsequentlyThe three numerator vectors have pairwise dot products $0$ and squared Euclidean norms $2,6,3$. Thus the displayed outputs are indeed an orthonormal basis, and the three nonzero remainders also confirm linear independence of the inputs. Keeping the removed coefficients produces the QR decomposition below.

An immediate reward is a reusable orthogonal projection: once an orthonormal basis of a vector subspace $W$ is known, any vector $b$ hasFor the linear span of just the first two inputs in the three-dimensional example, $b=(0,0,1{)}^T$ gives $P_{W}b=(1/3,-1/3,2/3{)}^T$. Its orthogonal projection residual is $(-1/3,1/3,1/3{)}^T=q_3/\sqrt{3}$, visibly in the missing orthogonal direction. This is the geometry behind the linear least-squares problem.

The formulas above describe exact arithmetic. With floating-point arithmetic, subtracting an orthogonal projection nearly equal to the input can leave a small remainder; dividing by its small norm can magnify existing roundoff error. The Classical Gram-Schmidt and Modified Gram-Schmidt arrangements agree in exact arithmetic but can give different rounded results. Exact linear dependence and a small computed remainder therefore require different judgments in a numerical implementation.^[ref]

A finite-dimensional linear subspace $W$ of an inner product space has a unique closest vector to any given vector $x$. This is its orthogonal projection $P_{W}x$. If $e_1,\dots ,e_d$ is an orthogonal basis of $W$, compute it one direction at a time:The inner product convention here is linear in its first argument, including over the complex numbers. For an orthonormal basis, each denominator is one. When $W$ consists only of the zero vector, the empty sum gives $P_{W}x=0$. The finite-dimensional formula is described in Francis Narcowich, Inner Products and Norms: minimization problems.

Take $W=\mathrm{span}\{(1,2{)}^T\}$ and $x=(3,1{)}^T$, using the dot product. With $e=(1,2{)}^T$,The orthogonal projection residual has $r\cdot e=2-2=0$. To verify the best approximation directly, every candidate in the line is $te$, andThe smallest squared Euclidean norm error is therefore $5$, attained only at $t=1$. The diagram connects the perpendicular residual to this one-variable quadratic polynomial.

The general mechanism is the Pythagorean theorem in an inner-product space. For any $w\in W$, the vectors $r=x-P_{W}x$ and $P_{W}x-w$ are orthogonal, soMoving away from $P_{W}x$ inside $W$ adds a nonnegative squared norm, which is zero only when $w=P_{W}x$. This proves both optimality and uniqueness.

The useful feature is separation of preparation from evaluation. Construct an orthonormal basis once with the Gram-Schmidt process, then obtain each new best approximation using inner products instead of a fresh optimization problem. When $W$ is the column space of a matrix, this gives linear least squares; when $W$ is spanned by orthogonal polynomials, it gives a least-squares polynomial in an orthogonal-polynomial basis. The ambient inner product space need not be a Hilbert space for this finite-dimensional construction. In an infinite-dimensional target linear subspace, existence requires additional hypotheses, supplied by orthogonal decomposition by a closed subspace.

An orthogonal projection residual is the difference $r=x-P_{W}x$ between a vector and its orthogonal projection onto a linear subspace $W$. It is orthogonal to every vector in $W$, so $⟨r,w⟩=0$ for all $w\in W$. The Gram-Schmidt process uses such residuals to remove directions already represented by earlier basis vectors.

This is an optimality condition as well as a geometric description. No permitted change within $W$ can remove a further orthogonal component of $r$. In the preceding dot product example, $r=(2,-1{)}^T$ is perpendicular to the line spanned by $(1,2{)}^T$.

The selected inner product matters. Keep $x=(3,1{)}^T$ and the same line, but use the weighted Euclidean inner product with $M=\mathrm{diag}(4,1)$. The projection coefficient and residual becomeNow $4(r_M{)}_1{e}_1+(r_M{)}_2{e}_2=5-5=0$, although $r_M\cdot e=-15/4\ne 0$. The orthogonality is with respect to $M$, so it need not look perpendicular in the ordinary drawing. The squared weighted error isThe weighted optimum reduces the first-coordinate error from $2$ to $5/4$, at the cost of increasing the magnitude of the second-coordinate error from $1$ to $5/2$. The weights express which errors are more costly.

In linear least squares, residual orthogonality to the column space is expressed by the normal equations for linear least squares. With a weighted Euclidean inner product, the corresponding real-matrix equation is $A^{T}M(x-Ac)=0$. This makes the choice of error measurement part of the mathematical model, rather than merely a plotting convention.

For any orthonormal basis $f,g$ of $U$, bilinearity givesTogether with $⟨f,g⟩=0$, these identities prove pairwise orthogonality. The construction subtracts the orthogonal projection onto a finite-dimensional subspace $U$ from $x^2$, leaving the orthogonal projection residual.

A reduced QR decomposition of an $m\times k$ matrix over the real numbers with full column rank, $m\ge k$, writes $A=QR$, where $Q$ is $m\times k$ with orthonormal columns and $R$ is a $k\times k$ upper triangular matrix with positive diagonal. Applying the Gram-Schmidt process to the columns $v_j$ of $A$ constructs $Q=(q_1\cdots q_k)$ and records the coefficientsThis last column identity is exactly the matrix multiplication $A=QR$. The upper triangular matrix records that the $j$th input uses only the first $j$ new directions, while its positive diagonal records the norms of the nonzero orthogonal projection residuals. Since $Q^{T}Q=I$, one also has $R=Q^{T}A$; $Q^T$ denotes the transpose and $I$ the identity matrix. Over the complex numbers, use the conjugate transpose $Q^{\ast }$ in these identities.

For the three-dimensional Gram-Schmidt process example, the exact matrices areFor example, the last column reconstructs asThe diagonal entries are approximately $1.414214,1.224745,1.154701$. Each is a norm of the new direction before it was scaled to length one.

For a linear least-squares problem, the orthogonal projection of $b$ onto the column space of $A$ is $Q{Q}^{T}b$. Hence the minimizing coefficients solve the linear system $Rx=Q^{T}b$ by back substitution. Let $A_2=(v_1{v}_2)$ consist of the first two columns of the example; its reduced factors are $Q_2=(q_1{q}_2)$ and the leading $2\times 2$ block $R_2$ of $R$. For $b=(0,0,1{)}^T$, the linear system $R_{2}x=Q_2^{T}b$ isThus $A_{2}x=(1/3,-1/3,2/3{)}^T$, agreeing with the orthogonal projection computed above, and the minimum Euclidean norm of $A_{2}x-b$ is $1/\sqrt{3}$. The QR decomposition separates building an orthonormal basis from recovering coefficients in the original basis.^[ref]

The vector projection multiplies the scalar projection by the chosen unit vector, giving the orthogonal projection onto the linear span of a nonzero real Euclidean vector $\mathbf{c}$:For $\mathbf{a}=(3,4)$ and $\mathbf{c}=(2,0)$, this gives $\mathbf{p}=(6/4)(2,0)=(3,0)$ and $\mathbf{r}=(0,4)$; the dot product $\mathbf{r}\cdot \mathbf{c}=0$ checks the orthogonal projection residual. Replacing $\mathbf{c}$ by $(-2,0)$ gives the same $\mathbf{p}=(-6/4)(-2,0)=(3,0)$: reversing direction changes the scalar projection, but preserves the vector projection onto the line.

For a short proof of the nearest-point property, any vector $\mathbf{z}$ on this line satisfies $\Vert \mathbf{a}-\mathbf{z}{\Vert }^2=\Vert \mathbf{r}{\Vert }^2+\Vert \mathbf{p}-\mathbf{z}{\Vert }^2$ by the Pythagorean identity. The second term is uniquely minimized at $\mathbf{z}=\mathbf{p}$. This connects a directional measurement to the orthogonal projection used in approximation.^[ref]

A symmetric positive-definite matrix $M$ defines an inner product on real coordinate vectors byFor diagonal $M$, positive diagonal entries weight the squared coordinate errors. This is the finite-coordinate version of a weighted inner product; the positive-definite matrix example appears in Francis Narcowich, Inner Products and Norms, standard inner products, example 8.

For $M=\mathrm{diag}(4,1)$, the squared norm of $(a,b{)}^T$ is $4a^2+b^2$. A first-coordinate error of one costs as much as a second-coordinate error of two. The unit sphere is the ellipse $4a^2+b^2=1$, and orthogonality means $4u_1{v}_1+u_2{v}_2=0$. Thus “closest” and “perpendicular” depend on the chosen inner product. The orthogonal projection residual example shows how this changes a best approximation.