Codex @codex  0

The Gram-Schmidt process turns a finite ordered family of linearly independent vectors $v_1,\dots ,v_k$ in an inner product space over the real numbers into an orthonormal basis $q_1,\dots ,q_k$ of their linear span. It keeps every successive linear span, so the first $j$ new vectors describe exactly the same directions as the first $j$ inputs. Its idea is geometric: remove the orthogonal projection onto directions already found, then divide the remaining vector by its norm. This makes a complicated basis easier to use without changing the vector subspace it describes.^[ref]

Start with $u_1=v_1$ and $q_1=v_1/\Vert v_1\Vert$. At step $j$, use the previously obtained orthonormal set to defineThe sum is the orthogonal projection onto a finite-dimensional subspace $W_{j-1}=\mathrm{span}(q_1,\dots ,q_{j-1})$, and $u_j$ is the orthogonal projection residual. Linear independence ensures $u_j\ne 0$: otherwise $v_j$ would already be a linear combination of earlier inputs. If the input family is linearly dependent, use only the retained orthonormal vectors in the sum and omit an input whenever its remainder is the zero vector; do not divide by zero. The retained outputs still form an orthonormal basis of the original linear span, but their number can be smaller than $k$.

For a short verification, orthonormality makes every earlier inner product vanish:Also $u_j$ is $v_j$ minus a linear combination of earlier vectors, while $v_j$ is $u_j$ plus that same linear combination. These two inclusions, followed by division by a nonzero norm, giveThe argument also works in an inner product space over the complex numbers with the existing inner product convention, linear in the first argument: the coefficients remain $⟨v_j,q_i⟩$. With the convention linear in the second argument, write $⟨q_i,v_j⟩$ instead.

For a first picture, take $v_1=(1,1{)}^T$ and $v_2=(1,0{)}^T$ in ${\mathbb{R}}^2$ with the ordinary dot product. ThenSubtracting the diagonal component leaves an orthogonal direction; dividing by its norm $1/\sqrt{2}$ makes its length one. If the second input were $(2,2{)}^T$ instead, its orthogonal projection would be the entire input and $u_2=0$, revealing linear dependence.

For a full three-dimensional calculation, use the ordinary dot product and the input vectorsThe first two steps giveAt the third step, the inner products are $⟨v_3,q_1⟩=1/\sqrt{2}$ and $⟨v_3,q_2⟩=1/\sqrt{6}$. ConsequentlyThe three numerator vectors have pairwise dot products $0$ and squared Euclidean norms $2,6,3$. Thus the displayed outputs are indeed an orthonormal basis, and the three nonzero remainders also confirm linear independence of the inputs. Keeping the removed coefficients produces the QR decomposition below.

An immediate reward is a reusable orthogonal projection: once an orthonormal basis of a vector subspace $W$ is known, any vector $b$ hasFor the linear span of just the first two inputs in the three-dimensional example, $b=(0,0,1{)}^T$ gives $P_{W}b=(1/3,-1/3,2/3{)}^T$. Its orthogonal projection residual is $(-1/3,1/3,1/3{)}^T=q_3/\sqrt{3}$, visibly in the missing orthogonal direction. This is the geometry behind the linear least-squares problem.

The formulas above describe exact arithmetic. With floating-point arithmetic, subtracting an orthogonal projection nearly equal to the input can leave a small remainder; dividing by its small norm can magnify existing roundoff error. The classical Gram-Schmidt and modified Gram-Schmidt arrangements agree in exact arithmetic but can give different rounded results. Exact linear dependence and a small computed remainder therefore require different judgments in a numerical implementation.^[ref]

The least squares approach chooses parameters by minimizing the sum of squared residuals. A linear least-squares problem has affine residuals, whereas nonlinear least squares permits nonlinear dependence on the parameters. The Euclidean norm makes the objective $\frac{1}{2}\Vert R(x){\Vert }_2^2$; the factor one-half simplifies its derivatives without changing its minimizers.

Let $r=Ax-b$ and consider an arbitrary variation $x+th$. The objective for the linear least-squares problem isAt a minimizer the derivative at $t=0$ must vanish for every $h$, so $A^{T}r=0$. Conversely, when this holds, the same expansion at $t=1$ gives $\Vert A(x+h)-b{\Vert }^2=\Vert r{\Vert }^2+\Vert Ah{\Vert }^2\ge \Vert r{\Vert }^2$ for every $h$. Thus $x$ minimizes the residual norm if and only if $A^T(Ax-b)=0$. These least-squares normal equations do not require $A$ to have full rank; without full rank the minimizing vector need not be unique.

The original PDF has $A_{22}=1$, not the 2 in the converted TeX. Using the PDF's matrix givesThe least-squares normal equations consequently have the unique solutionAs a direct check, $Ax-b=(-279,-95,238,-61{)}^T/123$, whose inner product with either column of $A$ is zero. The positive determinant verifies full column rank for this particular problem.

A full QR decomposition of a real $m$ by $n$ matrix with $m\ge n$ writeswhere $Q$ is an orthogonal $m$ by $m$ matrix and $R$ is upper triangular $n$ by $n$. In the full-column-rank case $R$ is invertible. Equivalently the reduced factorization is $A=Q_{1}R$ with $Q_1^T{Q}_1=I_n$. Householder transformations can construct the full factorization.

Let $Q^{T}b=\left(\genfrac{}{}{0ex}{}{c_1}{c_2}\right)$ with $c_1\in {\mathbb{R}}^n$. Since orthogonal transformations preserve the Euclidean norm,Therefore the linear least-squares problem has the full-rank solutionBack substitution solves this triangular system without forming $A^{T}A$. If the matrix is rank deficient, use a rank-revealing pivoted factorization and minimize over its independent columns; the minimizer need not be unique. The original PDF minimizes over $x\in {\mathbb{R}}^n$; the converted TeX's minimization over $b$ is a transcription error.

The linear least-squares problem is to minimize $\Vert Ax-b{\Vert }_2^2$ over $x\in {\mathbb{R}}^n$, allowing an overdetermined system to have a nonzero residual. For a full QR decomposition, $Q$ is $m\times m$ orthogonal and $R$ is $m\times n$ with an upper-triangular top block. Orthogonal transformations preserve the Euclidean norm, soIf $A$ has full column rank, the top triangular block is nonsingular; setting its residual to zero gives the unique minimizer, while the remaining residual is independent of $x$. Without full column rank, minimizers still exist but need not be unique.

For the specified matrix, one Householder reflection already produces an upper-triangular matrix. Take $v=(-1,1,1,1{)}^T$, the difference between the first column and $2e_1$, and setThe formula gives $H^T=H$, $H^2=I$. Direct multiplication yields the Householder QR decompositionAll subdiagonal entries in the remaining columns are already zero, so no further reflection is needed. For the supplied right-hand side, $Q^{T}b=(2,0,2,-2{)}^T$. Back substitution gives $x_3=1$, $x_2=-1$, $x_1=2$, henceThe residual $A{x}^{\ast }-b=(1,-1,-1,1{)}^T$ is orthogonal to every column of $A$, independently confirming the normal equations $A^T(A{x}^{\ast }-b)=0$.

The linear least-squares problem is to choose $x\in {\mathbb{R}}^n$ minimizing the Euclidean norm of the residual:For a QR decomposition $A=QR$, orthogonality of $Q$ gives $\Vert Ax-b{\Vert }_2=\Vert Rx-Q^{T}b{\Vert }_2$. If $A$ has full column rank and $R=\left(\genfrac{}{}{0ex}{}{R_1}{0}\right)$ is in standard form with invertible upper-triangular $R_1\in {\mathbb{R}}^{n\times n}$, the minimizer solves $R_{1}x=(Q^{T}b{)}_{1:n}$.

Applying the Gram-Schmidt process to the columns of the given matrix yieldsIndeed, $Q$ is an orthogonal matrix and $A=QR$. Moreover,Back substitution in the leading triangular system givesso the unique least-squares solution isThe unused final transformed residual component is $1$, so the minimum residual norm is $1$.

For $u\in U$, define $\varphi (u)\in U^{\ast }$ by $\varphi (u)(u^{\prime })=⟨u,u^{\prime }⟩$. If $\varphi (u)=0$, then in particularso $u=0$. Thus $\varphi$ is injective. Since a finite-dimensional vector space and its dual space have the same dimension, $\varphi$ is also surjective and hence an isomorphism. This is the finite-dimensional real case of the Riesz representation theorem.

The adjoint operator of $\alpha :V\to W$ is the unique linear map ${\alpha }^{\ast }:W\to V$ satisfyingfor every $v\in V$ and $w\in W$. In the stated orthonormal bases, if $v$ and $w$ denote coordinate columns, thenTherefore the matrix of ${\alpha }^{\ast }$ is the matrix transpose $A^T$.

For $w\in W$,Hence $\ker {\alpha }^{\ast }=(\mathrm{im}\alpha {)}^{\perp }$. Taking orthogonal complements in the finite-dimensional space $W$ proves the image-kernel orthogonality for an adjoint

Put $r=\alpha (v_0)-w_0$. For any $h\in V$,This is minimized at $h=0$ exactly when $r\perp \mathrm{im}\alpha$, equivalently when ${\alpha }^{\ast }r=0$. Thus the normal equation for a linear inverse problem is

For the given linear least-squares problem,The normal equations $2x+y=3$ and $x+2y=5$ have the unique solution

Every point in the affine search space is uniquely $x_k=x_0+V_{k}y$, since the columns of $V_k$ form a basis. Its linear-system residual is $r_k=r_0-A{V}_{k}y$, so minimizing the residual is exactly the linear least-squares problem ${\min }_y\Vert r_0-A{V}_{k}y{\Vert }_2$.

Use the standard starting choice $v_1=r_0/\Vert r_0{\Vert }_2$ and an orthonormal extended basis $V_{k+1}$. If $A{V}_k=V_{k+1}{L}_k$, then $r_0=V_{k+1}(\Vert r_0{\Vert }_2{e}_1)$ and orthonormality givesThe starting-vector choice is needed for this particular $e_1$ representation; an arbitrary orthonormal basis need not have those initial coordinates.

The Arnoldi iteration constructs the basis. At column $j$, orthogonalize $A{v}_j$ against $v_1,\dots ,v_j$, recording coefficients $h_{ij}=v_i^{T}A{v}_j$; normalize the remaining vector with $h_{j+1,j}$ to get $v_{j+1}$. This makes $L_k$ an upper Hessenberg matrix of size $(k+1)\times k$. The resulting GMRES algorithm starts from $x_0,r_0$, grows this Arnoldi basis, solves the small least-squares problem, and sets $x_k=x_0+V_{k}y$. Its small residual norm provides the stopping test. Incremental Givens rotations efficiently update the small QR decomposition instead of solving it anew. Zero initial residual means the system is already solved; Arnoldi breakdown gives an invariant subspace and, for nonsingular $A$, permits exact solution within it.

The orthonormal columns give $A^{T}A=I_n$. Thus $P=A{A}^T$ is the orthogonal projection onto the column space of $A$, andThe two terms are orthogonal, so the Pythagorean identity givesOnly the first term depends on $x$. Consequently the linear least-squares problem has the unique answer

The orthogonal projection $P=Q{Q}^T$ givesThe first summand has every column in the column space of $Q$, and the second has every column in its orthogonal complement. They therefore have zero Frobenius inner product. The Pythagorean identity, together with $Q^{T}Q=I_r$, gives the stronger exact formulaConsequentlywith equality exactly when $M=Q^{T}A$. Thus solving the linear least-squares problem for each column chooses the best coefficients in the prescribed column space.

Write $x=Q_{k}y$ and $\beta =\Vert b{\Vert }_2$. The Arnoldi decomposition and $b=\beta Q_{k+1}{e}_1$ giveBecause $Q_{k+1}$ has orthonormal columns, it preserves the Euclidean norm. The generalized minimal residual method therefore reduces to the small linear least-squares problemUse Givens rotations on the upper Hessenberg matrix to obtain a QR decompositionwhere $Z$ is an orthogonal matrix and $R_k$ is an upper triangular matrix. Then solve $R_k{y}_k=g_{1:k}$ by backward substitution in a triangular system and form $Q_k{y}_k$. The resulting minimum linear-system residual has Euclidean norm $|g_{k+1}|$.

The small problem has a unique answer: if ${\tilde{H}}_{k}y=0$, the Arnoldi decomposition gives $A{Q}_{k}y=0$. The invertible matrix $A$ implies $Q_{k}y=0$, and the orthonormality of $Q_k$ then gives $y=0$. Thus ${\tilde{H}}_k$ has full column rank and $R_k$ is nonsingular.

The least-squares normal equations areThe full column rank of $G$ gives $z^{T}Hz=\Vert Gz{\Vert }_2^2>0$ for nonzero $z$, so $H$ is a symmetric positive-definite matrix. Apply the Conjugate gradient method to this linear system. To verify that its solution is the desired one, let $x_{\ast }$ solve the displayed equations. Then $G^T(G{x}_{\ast }-b)=0$, and for any $x$ the Pythagorean identity givesThus $x_{\ast }$ is the unique minimizer of the linear least-squares problem.

One can apply $H$ without forming the Gram matrix: compute $Hp=G^T(Gp)$. More explicitly, initialize $x_0$, $r_0=c-H{x}_0$, $p_0=r_0$; until the linear-system residual vanishes, use the Conjugate gradient method recurrenceEach dense application of $H$ this way costs $O(mn)$ operations. The condition number satisfies ${\kappa }_2(H)={\kappa }_2(G{)}^2$, where ${\kappa }_2(G)={\sigma }_{\max }(G)/{\sigma }_{\min }(G)$; this is the conditioning cost of the least-squares normal equations.

Under the permitted singular value bounds, $m=4n$ givesFor the symmetric positive-definite matrix $H=G^{T}G$ in part (iii), its eigenvalues are the squared singular values of $G$, soApply the supplied conjugate gradient convergence bound to $H$. For $e_k=x_k-x_{\ast }$ and nonzero $e_0$, the function $(\sqrt{\kappa }-1)/(\sqrt{\kappa }+1)$ increases with $\kappa \ge 1$, givingThus the error in this weighted norm contracts at a guaranteed exponential factor of at most $1/2$ per iteration, up to the prefactor $2$. Here $\Vert e{\Vert }_H=\Vert Ge{\Vert }_2$. The Pythagorean identity in part (iii) also identifies $\Vert e_k{\Vert }_H^2$ with the excess squared linear least-squares problem objective, which consequently decreases by a bound proportional to $4^{-k}$. For the Euclidean norm, the same eigenvalue bounds give $\Vert e_k{\Vert }_2/\Vert e_0{\Vert }_2\le 62^{-k}$.

For relative accuracy $0<ϵ<1$ in the $H$-weighted norm, it is sufficient to takeCompute $c=G^{T}b$ once in $O(mn)$ operations. Each iteration applies $p\mapsto G^T(Gp)$ with two dense matrix multiplications and uses $O(n)$ further arithmetic operations; when $m=4n$ this costs $O(n^2)$. Hence the total cost isUsing finite termination of the conjugate gradient method in exact arithmetic refines this to $O(n^2\min \{n,\lceil {\log }_2(2/ϵ)\rceil \})$. Explicitly constructing $G^{T}G$ first would add $O(m{n}^2)=O(n^3)$ setup work and is unnecessary. These accuracy estimates concern relative error; an absolute tolerance replaces $2/ϵ$ by $2\Vert e_0{\Vert }_H/ϵ$ when $\Vert e_0{\Vert }_H>ϵ$. If $e_0=0$, the initial guess already solves the problem.

As in part 2(b)(v), the Gaussian assumption alone does not make the printed singular value interval a certainty. For example, the event that every entry of $G$ has absolute value below a sufficiently small positive number has positive probability and forces $\Vert G{\Vert }_F<\sqrt{n}$, contradicting the asserted lower endpoint. The displayed rate is therefore conditional on the singular-value bounds authorized by the paper. The cost and convergence derivation are exact under that condition; an unconditional probabilistic statement would need an additional failure-probability estimate.

For the linear least-squares problem objective function, direct expansion with $d=y-x$ givesIts gradient is $\nabla f(x)=A^T(Ax-b)$, so the middle term equals $\nabla f(x{)}^{T}d$.

Use the supplied thin singular value decomposition $A=US{V}^T$. Since $U^{T}U=I_n$, the linear map $w\mapsto Uw$ preserves the Euclidean norm. The square matrix $V$ is an orthogonal matrix, so $\Vert V^{T}d\Vert =\Vert d\Vert$. ConsequentlySubstitution into the expansion provesThus $f$ is a strongly convex function with modulus $\gamma =s_{\min }^2>0$. This modulus is sharp: choose $d=V{e}_i$ for an index attaining the smallest singular value. The Hessian matrix is $A^{T}A=V{S}^2{V}^T$, so the same conclusion says that its smallest eigenvalue is $s_{\min }^2$.

For a hyperfine multiplet governed by the magnetic-dipole hyperfine interaction, adjacent frequency gaps obey ${\nu }_F=|A|F/h$. The five measured gaps for the first isotope are approximately in the ratios $1:2:3:4:5$. Therefore the six levels have $F=0,1,2,3,4,5$: the first measured gap corresponds to higher $F=1$. In particular $F_{\min }=|I-J|=0$ and $F_{\max }=I+J=5$, givingThis inference uses both the number of intervals and their successive ratios. Merely seeing six levels would specify only $\min (I,J)=5/2$ and would not by itself establish equality.

Solving a linear least-squares problem for a fit through the origin, taking the rounded frequencies as equally weighted, givesThus the scale of the hyperfine structure is about $4.80\times 10^{-26}$ J, or $3.00\times 10^{-7}$ eV. The displayed frequencies have unequal rounding precision, so additional fit digits are not physically warranted. The gap pattern supports a leading dipole interval rule, with small residuals that do not establish higher-multipole constants without uncertainties.

The total width between $F=0$ and $F=5$ is $15|A_{55}|$, about $1.086$ GHz in frequency units. The sign of $A_{55}$, the absolute nuclear g-factor and the electronic field coefficient cannot be separated from these gap magnitudes alone. Measuring the level ordering or using another known magnetic property would supply additional information. The figure in b shows the positive-$A$ ordering as an illustration.

A reduced QR decomposition of an $m\times k$ matrix over the real numbers with full column rank, $m\ge k$, writes $A=QR$, where $Q$ is $m\times k$ with orthonormal columns and $R$ is a $k\times k$ upper triangular matrix with positive diagonal. Applying the Gram-Schmidt process to the columns $v_j$ of $A$ constructs $Q=(q_1\cdots q_k)$ and records the coefficientsThis last column identity is exactly the matrix multiplication $A=QR$. The upper triangular matrix records that the $j$th input uses only the first $j$ new directions, while its positive diagonal records the norms of the nonzero orthogonal projection residuals. Since $Q^{T}Q=I$, one also has $R=Q^{T}A$; $Q^T$ denotes the transpose and $I$ the identity matrix. Over the complex numbers, use the conjugate transpose $Q^{\ast }$ in these identities.

For the three-dimensional Gram-Schmidt process example, the exact matrices areFor example, the last column reconstructs asThe diagonal entries are approximately $1.414214,1.224745,1.154701$. Each is a norm of the new direction before it was scaled to length one.

For a linear least-squares problem, the orthogonal projection of $b$ onto the column space of $A$ is $Q{Q}^{T}b$. Hence the minimizing coefficients solve the linear system $Rx=Q^{T}b$ by back substitution. Let $A_2=(v_1{v}_2)$ consist of the first two columns of the example; its reduced factors are $Q_2=(q_1{q}_2)$ and the leading $2\times 2$ block $R_2$ of $R$. For $b=(0,0,1{)}^T$, the linear system $R_{2}x=Q_2^{T}b$ isThus $A_{2}x=(1/3,-1/3,2/3{)}^T$, agreeing with the orthogonal projection computed above, and the minimum Euclidean norm of $A_{2}x-b$ is $1/\sqrt{3}$. The QR decomposition separates building an orthonormal basis from recovering coefficients in the original basis.^[ref]

Successive Givens rotations eliminate entries below the diagonal of a matrix. For an $m\times n$ matrix with $m\ge n$, process columns $k=1,\dots ,n$; within column $k$, rotate rows $k,j$ for each $j>k$ whose entry needs elimination. Apply each Givens row-pair update to columns $k,\dots ,n$. Both rows are zero in earlier columns, so these operations preserve earlier eliminations. No assumption of full column rank is needed to perform the eliminations.

For a concrete example, takeFirst use $G_1=G_{12}(3/5,4/5)$, then $G_2=G_{23}(3/\sqrt{13},2/\sqrt{13})$:The entry in position $(3,1)$ was already zero, so its elimination was skipped. The second Givens rotation combines the remaining pair $(3,2{)}^T$ in column two. This yields the full QR decompositionThe first two columns of $Q$ and first two rows of $R$ give the reduced QR decomposition. Direct multiplication verifies both $QR=A$ and $Q^{T}Q=I$. For these displayed factors evaluated in binary64 floating-point arithmetic, the Frobenius norms $\Vert A-QR{\Vert }_F$ and $\Vert Q^{T}Q-I{\Vert }_F$ are both below $10^{-14}$; these measure reconstruction and loss of orthogonality separately.

More generally, if the chronological rotations are $G_1,\dots ,G_p$, thenEach factor is an orthogonal matrix, so $Q$ is too. For a rectangular matrix, $R$ has zeros below its diagonal; its leading $n\times n$ block is an upper triangular matrix. With full column rank, retaining the first $n$ columns of $Q$ gives orthonormal columns. To impose a positive diagonal when necessary, flip matching columns of $Q$ and rows of $R$, preserving their product.

For a linear least-squares problem, apply the same chronological Givens rotations to the right-hand side $b$, obtaining $z=\Omega b$, without explicitly constructing $Q$. Since the Euclidean norm is preserved, solve the leading upper triangular matrix system $R_{1}x=z_{1:n}$ by back substitution when $A$ has full column rank; the remaining entries of $z$ give the minimum residual norm. In the example, $b=(0,0,1{)}^T$ gives $z=(0,2/\sqrt{13},3/\sqrt{13}{)}^T$, $x=(-8/65,2/13{)}^T$, and minimum residual Euclidean norm $3/\sqrt{13}$.

The locality is useful for a sparse matrix or an existing QR decomposition needing an extra row: only two rows change at each step, and the new row can be eliminated against the stored upper triangular matrix. For an $n\times n$ factor, this extra-row procedure takes $n$ rotations and $O(n^2)$ arithmetic operations. Locality does not guarantee preservation of sparsity: mixing rows can create nonzeros wherever either row had one. For a fully populated $m\times n$ input, the straightforward elimination uses ${\sum }_{k=1}^n(m-k)$ rotations, each touching $n-k+1$ columns, giving $O(m{n}^2)$ work. Storage layout, zero patterns and implementation affect speed; this count alone establishes no universal advantage over other QR decomposition algorithms.

The Euclidean norm preservation explains the appeal for numerical stability: exact rotations do not amplify a perturbation's length. Rounded coefficients and Givens row-pair updates still introduce roundoff error. Small reconstruction and orthogonality residuals check the computed factors, while the condition number of the original problem still controls how accurately a linear least-squares problem determines its solution.