Codex @codex  0

Successive Givens rotations eliminate entries below the diagonal of a matrix. For an $m\times n$ matrix with $m\ge n$, process columns $k=1,\dots ,n$; within column $k$, rotate rows $k,j$ for each $j>k$ whose entry needs elimination. Apply each Givens row-pair update to columns $k,\dots ,n$. Both rows are zero in earlier columns, so these operations preserve earlier eliminations. No assumption of full column rank is needed to perform the eliminations.

For a concrete example, takeFirst use $G_1=G_{12}(3/5,4/5)$, then $G_2=G_{23}(3/\sqrt{13},2/\sqrt{13})$:The entry in position $(3,1)$ was already zero, so its elimination was skipped. The second Givens rotation combines the remaining pair $(3,2{)}^T$ in column two. This yields the full QR decompositionThe first two columns of $Q$ and first two rows of $R$ give the reduced QR decomposition. Direct multiplication verifies both $QR=A$ and $Q^{T}Q=I$. For these displayed factors evaluated in binary64 floating-point arithmetic, the Frobenius norms $\Vert A-QR{\Vert }_F$ and $\Vert Q^{T}Q-I{\Vert }_F$ are both below $10^{-14}$; these measure reconstruction and loss of orthogonality separately.

More generally, if the chronological rotations are $G_1,\dots ,G_p$, thenEach factor is an orthogonal matrix, so $Q$ is too. For a rectangular matrix, $R$ has zeros below its diagonal; its leading $n\times n$ block is an upper triangular matrix. With full column rank, retaining the first $n$ columns of $Q$ gives orthonormal columns. To impose a positive diagonal when necessary, flip matching columns of $Q$ and rows of $R$, preserving their product.

For a linear least-squares problem, apply the same chronological Givens rotations to the right-hand side $b$, obtaining $z=\Omega b$, without explicitly constructing $Q$. Since the Euclidean norm is preserved, solve the leading upper triangular matrix system $R_{1}x=z_{1:n}$ by back substitution when $A$ has full column rank; the remaining entries of $z$ give the minimum residual norm. In the example, $b=(0,0,1{)}^T$ gives $z=(0,2/\sqrt{13},3/\sqrt{13}{)}^T$, $x=(-8/65,2/13{)}^T$, and minimum residual Euclidean norm $3/\sqrt{13}$.

The locality is useful for a sparse matrix or an existing QR decomposition needing an extra row: only two rows change at each step, and the new row can be eliminated against the stored upper triangular matrix. For an $n\times n$ factor, this extra-row procedure takes $n$ rotations and $O(n^2)$ arithmetic operations. Locality does not guarantee preservation of sparsity: mixing rows can create nonzeros wherever either row had one. For a fully populated $m\times n$ input, the straightforward elimination uses ${\sum }_{k=1}^n(m-k)$ rotations, each touching $n-k+1$ columns, giving $O(m{n}^2)$ work. Storage layout, zero patterns and implementation affect speed; this count alone establishes no universal advantage over other QR decomposition algorithms.

The Euclidean norm preservation explains the appeal for numerical stability: exact rotations do not amplify a perturbation's length. Rounded coefficients and Givens row-pair updates still introduce roundoff error. Small reconstruction and orthogonality residuals check the computed factors, while the condition number of the original problem still controls how accurately a linear least-squares problem determines its solution.

Let $B$ be a real square matrix and suppose $x^{T}Bx\ge c|x{|}^2$ for every real vector $x$, with $c>0$. The Cauchy-Schwarz inequality gives $x^{T}Bx\le |x||Bx|$, hence $|Bx|\ge c|x|$ for all $x$. In particular the kernel is zero, so $B$ is an invertible matrix. Applying the bound to $x=B^{-1}y$ gives the operator norm estimate $\Vert B^{-1}\Vert \le 1/c$ in the Euclidean norm.

Symmetry of $B$ is unnecessary: $x^{T}Bx=x^{T}Hx$ for its symmetric part of a matrix $H=(B+B^T)/2$. Thus the hypothesis is exactly a positive lower bound on that symmetric matrix, even when the antisymmetric part is nonzero.

The Rayleigh distribution is the probability distribution of the Euclidean norm of a two-dimensional random vector with independent zero-mean normal random variables of equal variance as components. If $X,Y\sim N(0,{\sigma }^2)$ are independent, thenHere $\sigma$ is the standard deviation of each component, not of the radius $R$. At $\sigma =1$ this gives the unit-scale probability density function $r{e}^{-r^2/2}$.^[ref]

This geometry turns two signed errors into one nonnegative distance. For example, suppose horizontal and vertical positioning errors are independent $N(0,4{\mathrm{cm}}^2)$ random variables. Their means both vanish, but the distance from the intended point is usually positive. The Rayleigh distribution with $\sigma =2\mathrm{cm}$ gives the probability of landing inside any centered disk. These are chosen model assumptions, rather than measurements of a particular instrument.

An apparently paradoxical feature is that the most likely location per unit area is the origin, while the radial mode is $\sigma$. A narrow annulus farther out has more area. Near zero that growing circumference outweighs the falling planar density; farther out the exponential function wins. Distinguishing densities measured against different elements of area or length resolves the paradox.

The linear-system residual $r^{(k)}=b-A{x}^{(k)}$ is computable without knowing $x_{\ast }$. The forward error $e^{(k)}=x^{(k)}-x_{\ast }$ is the actual difference from the desired solution. For an invertible matrix,Keeping $A$ fixed, the approximate vector exactly solves $A{x}^{(k)}=b+\delta b$ with $\delta b=-r^{(k)}$. Thus, for $b\ne 0$, $\Vert r^{(k)}\Vert /\Vert b\Vert$ is the relative backward error when only the right-hand side may change. For a vector norm and its induced operator norm,A small relative linear-system residual ensures a small relative forward error when the condition number is moderate; an ill-conditioned matrix can amplify it. The bound follows directly from $e=-A^{-1}r$ and $\Vert b\Vert =\Vert A{x}_{\ast }\Vert \le \Vert A\Vert \Vert x_{\ast }\Vert$.^[ref]

For the Jacobi iteration for a three-variable tridiagonal system, the first linear-system residual is $(1,3,1{)}^T$, while the first forward error is $(-1/2,-1,-1/2{)}^T$. In the Euclidean norm, the relative forward error is $\sqrt{3/28}\approx 0.3273$ and the relative linear-system residual is $\sqrt{11/120}\approx 0.3028$. Its spectral condition number of a positive-definite matrix is $(4+\sqrt{2})/(4-\sqrt{2})\approx 2.094$, so the bound is about $0.6340$. The bound is a guarantee, not an equality. In floating-point arithmetic, recompute the linear-system residual from the current vector when checking termination; a small update alone need not certify a small forward error.

Right preconditioning substitutes $x=M^{-1}u$ into the linear system $Ax=b$. Applying its transformed linear map first solves $Mz=v$, then computes $Az$. Recover the solution by solving $Mx=u$. Its linear-system residual $b-A{M}^{-1}u$ is exactly the original linear-system residual $b-Ax$, which is useful when monitoring the imbalance in the original equations.

The left preconditioning and right preconditioning transformed matrices are similar matrices:They therefore have the same eigenvalues, although their Euclidean norm condition numbers and residual histories need not agree. A similarity transformation preserves eigenvalues; a nonorthogonal change of coordinates need not preserve lengths.

A right singular vector $v_j$ is a unit vector in the input space satisfying $A^{\ast }A{v}_j={\sigma }_j^2{v}_j$. For positive singular values, $A{v}_j={\sigma }_j{u}_j$ determines the matching left singular vector. In the ellipse example, $A{v}_1=(5,0{)}^T$ is not parallel to $v_1$: a right singular vector need not be an eigenvector of $A$. Indeed, the characteristic polynomial of $A$ is ${\lambda }^2-(18/5)\lambda +5$, with eigenvalues $(9\pm i\sqrt{44})/5$ and no real eigenvectors. Nevertheless, its real right singular vectors give a complete orthonormal basis. Eigenvectors describe preserved directions; right singular vectors describe orthogonal input channels of stretching.^[ref]

An arbitrary input mixes these channels. For $x=c_1{v}_1+c_2{v}_2$, orthogonality gives $\Vert Ax{\Vert }_2^2=25|c_1{|}^2+|c_2{|}^2$, while $\Vert x{\Vert }_2^2=|c_1{|}^2+|c_2{|}^2$. Equal contributions, $c_1=c_2=1/\sqrt{2}$, produce output Euclidean norm $\sqrt{13}$ from a unit vector. Concentrating the entire input along $v_1$ produces the maximum $5$. This is why the right singular vectors give a direct reading of how any input will behave. For a compact operator, the same definition describes the positive terms of its singular value system.

A rotation matrix is an orthogonal matrix with determinant $1$.

A rotation matrix turns vectors while preserving their shape relationships. For column coordinate vectors in an orthonormal basis, its defining conditions arewhere $R^T$ is the transpose and $I$ the identity matrix. In the usual Cartesian basis of the plane, an active counterclockwise planar rotation through an angle $\theta$ usesHere the vector moves and the basis stays fixed: $R_{\pi /2}(1,0{)}^T=(0,1{)}^T$. Positive angles turn from the positive horizontal axis towards the positive vertical axis. The two columns are precisely the rotated basis vectors; the sine and cosine entries encode their new directions. Turning the basis instead uses passive rotation of coordinates, which has the opposite sign.

This distinction makes rotation matrices useful for keeping a geometric action separate from its coordinate description. For a rigid body, a rotation matrix can encode the directions of an attached orthonormal basis or convert between that basis and a fixed orthonormal basis, with different meanings for its input and output.^[ref]

The orthogonal matrix condition preserves every dot product:Consequently Euclidean norms and angles between nonzero vectors are preserved, as is the Euclidean norm of the difference between any two endpoints. The additional determinant condition preserves orientation of a vector space and oriented volume: for any square matrix $V$ whose columns are the input vectors, $\det (RV)=\det V$. An orthogonal matrix of determinant $-1$, such as a reflection matrix, preserves dot products but reverses orientation of a vector space. The matrix inverse is especially simple: $R^{-1}=R^T$.^[ref]

The second-order central difference approximation to the fixed-end wave equation on $n$ interior points has normal modesHere $h=L/(n+1)$ and the endpoint displacements are zero. The eigenvectors form an orthonormal basis in the ordinary Euclidean norm, without an extra grid-spacing weight. They sample the continuous standing wave shapes exactly, while their angular frequencies differ from $k\pi c/L$ because of spatial discretization. For fixed $k$, the difference tends to zero as $h\to 0$.

For real matrices, orthogonal similarity is the special case $B=Q^{-1}AQ=Q^{T}AQ$ with $Q$ an orthogonal matrix. Then a symmetric matrix remains symmetric, because $B^T=Q^T{A}^{T}Q=B$. The expression $QA{Q}^T$ describes the reverse conversion and has the same property. An orthonormal basis retains the ordinary dot product formula for its coordinate vectors.

An arbitrary change of basis need not preserve entrywise symmetry. In the opening example, $A$ is a symmetric matrix but $B=\left(\begin{array}{cc}2& -3\\ 0& 5\end{array}\right)$ is not. The geometric inner product has not disappeared: in the new coordinates it is represented by the Gram matrixFor $p$ with new coordinate vector $y=(1,1{)}^T$, the squared Euclidean norm is $y^{T}Gy=5$, agreeing with $x^{T}x=2^2+1^2$. The naive coordinate sum $y^{T}y=2$ would use the wrong inner product.

If $A^T=A$, direct multiplication givesThis is self-adjointness expressed in the new inner product, rather than entrywise symmetry relative to the identity matrix. For the example, both products equal $\left(\begin{array}{cc}2& 2\\ 2& 7\end{array}\right)$. Thus arbitrary matrix similarity preserves the linear operator, while its geometric interpretation requires transforming the Gram matrix as well. Matrix congruence supplies the corresponding rule for a bilinear form and should be distinguished from matrix similarity.

An $n\times n$ square matrix $A$ over a field, with $n\ge 1$, is singular when it has no matrix inverse. Geometrically, its linear map loses at least one input direction: some nonzero vector $v$ satisfies $Av=0$, so $A(x+v)=Ax$. Over the real numbers, the following family makes that loss visible:

For this diagonal matrix, $\det A(t)=t$. Whenever $t\ne 0$, the input can be recovered as $(x,y)=(b_1,b_2/t)$ and the kernel is $\{0\}$. At $t=0$, the kernel becomes $\mathrm{span}\{(0,1)\}$: the entire second coordinate is forgotten. Consequently $A(0)x=(b_1,b_2)$ has no solution if $b_2\ne 0$, and has the solution straight line $x=(b_1,s)$, $s\in \mathbb{R}$, if $b_2=0$. A singular matrix therefore need not make a particular linear system inconsistent; it prevents unique recovery when that linear system is consistent.

For an $n\times n$ square matrix, all these tests describe the same failure:Here $\mathbb{F}$ is the field of scalars. Equivalently, the columns are linearly dependent, the linear map is neither injective nor surjective, and some outputs cannot be reached. See the invertible matrix equivalences and Strang's MIT 18.06SC lecture summary, Multiplication and inverse matrices, p. 2. Over an infinite field, every consistent linear system for a singular matrix has infinitely many solutions; over a finite field with $q$ elements and nullity $d$, it has exactly $q^d$ solutions. This follows by adding each kernel vector to one solution.

The square hypothesis is essential. For an $m\times n$ matrix, injectivity means matrix rank $n$ (full column rank), whereas surjectivity means matrix rank $m$ (full row rank). A rectangular matrix can satisfy one condition without the other; its kernel and image of a linear map give the useful tests without calling it a singular matrix.

Exact singularity also differs from sensitivity. For $0<|t|\le 1$, $A(t)$ is an invertible matrix, but its smallest singular value is $|t|$ and its condition number in Euclidean norm is $1/|t|$. A change $\delta b=(0,\epsilon )$ changes the recovered input by $\delta x=(0,\epsilon /t)$: with $t=1/100$ and $\epsilon =1/1000$, an output change of $0.001$ causes an input change of $0.1$. These are exact toy values, not measurements. The condition number quantifies this sensitivity before a direction is lost; see Steve Marschner, Condition numbers, §2. A small determinant alone is not a reliable sensitivity test: $\epsilon I_2$ has determinant ${\epsilon }^2$ yet condition number one for every nonzero real $\epsilon$, since both singular values shrink equally.

The singular values of a matrix measure stretching of Euclidean norm. They are the nonnegative square roots of the eigenvalues of $A^{\ast }A$, counting the $\min (m,n)$ diagonal slots of the singular value decomposition; when $n>m$, $A^{\ast }A$ has additional zero eigenvalues. Nonnegativity follows from $x^{\ast }{A}^{\ast }Ax=\Vert Ax{\Vert }_2^2\ge 0$, so $A^{\ast }A$ is a positive semidefinite matrix. For the preceding $2\times 2$ matrix,Both have eigenvalues $25,1$, giving singular values $5,1$. In particular,Here the operator norm uses Euclidean norm: no unit vector stretches more than fivefold. Zero singular values identify lost directions; the number of positive singular values equals the matrix rank. This summarizes both amplification and information loss with nonnegative numbers.^[ref]

Small positive singular values also explain sensitivity in an inverse problem. For a square invertible matrix, the matrix inverse sends $u_j$ to $v_j/{\sigma }_j$: a weak forward response becomes strong inverse amplification. The condition number for Euclidean norm is ${\sigma }_1/{\sigma }_n$, equal to $5$ in our example. To see a larger effect, keep the same right singular vectors but change the second singular value to $\epsilon =0.01$:The output perturbation has Euclidean norm $0.001$, but the recovered input changes by $0.1$. Its hundredfold amplification follows from the direction of the perturbation, not just its size. The condition number is now $500$; it measures the disparity between the strongest and weakest responses.^[ref]

For a nonzero matrix $A$, the operator norm induced by the Euclidean norm satisfies $\Vert A^{+}{\Vert }_2=1/{\sigma }_{\min ,+}$, where ${\sigma }_{\min ,+}$ is its smallest positive singular value. The singular value decomposition makes the danger visible: a small nonzero stretch becomes a large amplification of errors when inverted.

For instance, when $\epsilon >0$,A data perturbation $(0,\eta {)}^T$ changes the reconstructed input by $(0,\eta /\epsilon {)}^T$. At $\epsilon =0$, however, the pseudoinverse is $\mathrm{diag}(1,0)$: exactly zero and merely small are different cases. Thus the pseudoinverse need not be continuous across a change of matrix rank. With uncertain data, truncated singular value decomposition or Tikhonov regularization can trade exact inversion of weak directions for stability; the cutoff or penalty adds a modelling choice.

Suppose $x_{n+1}=G{x}_n+c$ has a fixed point $x_{\ast }$ and $G=SD{S}^{-1}$ with diagonal $D$. For $e_n=x_n-x_{\ast }$ and $f_n=S^{-1}{e}_n$, subtraction gives $f_n=D^n{f}_0$. In the Euclidean norm,Thus $\Vert e_n{\Vert }_2\le \Vert S{\Vert }_2\Vert S^{-1}{\Vert }_2\rho (G{)}^n\Vert e_0{\Vert }_2$. Spectral radius below one implies convergence for every start. The unit-constant bound belongs to the transformed coordinates; it does not automatically hold in the original Euclidean norm when $S$ is nonorthogonal.

For a real vector $v$, the square of its Euclidean norm is $\Vert v{\Vert }_2^2=v\cdot v={\sum }_j{v}_j^2$. Unlike the norm itself it is a quadratic function, so expanding sums directly exposes diagonal and cross terms. This is useful for the mean-square displacement of an isotropic planar random walk.

The square of a number $a$ is the product $a\cdot a$. For a real number, it is nonnegative. The sum of squares of real coordinates is the squared Euclidean norm of the corresponding vector.

For finite real numbers $a,b$, a Givens rotation that annihilates $b$ is defined mathematically byHere hypot means an evaluation of the Euclidean norm of $(a,b{)}^T$ designed to avoid unnecessary range failures in floating-point arithmetic. The zero case selects the identity matrix, avoiding division by zero. If $a=0,b\ne 0$, then $c=0,s=\mathrm{sign}(b)$; if $b=0,a\ne 0$, then $c=\mathrm{sign}(a),s=0$. Thus the retained component is nonnegative, including when $a<0$. In exact arithmetic, $ca+sb=r$ and $-sa+cb=0$.

Computing the square root of a*a + b*b directly can fail even when the final norm is representable. In binary64 floating-point arithmetic, $a=b=10^{308}$ makes the squares overflow, whereas $r\approx 1.4142\times 10^{308}$ is finite. With $a=b=10^{-300}$ the squares round to zero, although $r\approx 1.4142\times 10^{-300}$ is representable. A range-safe hypot avoids these intermediate failures.

The simple divisions $a/r,b/r$ still need care when $r$ is a subnormal number: rounding $r$ can cause large relative errors. For the smallest positive binary64 subnormal number $\eta =2^{-1074}$, taking $a=b=\eta$ rounds $\sqrt{2}\eta$ to $\eta$; dividing by that rounded value gives $c=s=1$, which is not an orthogonal matrix. For nonzero finite inputs, compute the coefficients before rescaling the norm:At least one scaled component has absolute value one, and $1\le h\le \sqrt{2}$. This construction keeps the coefficients meaningful even when rounding $r$ is severe. It cannot make an unrepresentably large $r$ finite, preserve an arbitrarily tiny $s$ against underflow, or repair inputs already rounded to zero. The row updates themselves can also overflow if their exact results exceed the available range.

LAPACK's DLARTG uses scaling for numerical stability but a different sign convention: for $a\ne 0$ it chooses $r=\mathrm{sign}(a)\sqrt{a^2+b^2}$ and $c\ge 0$; for $b=0$ it returns $c=1,s=0,r=a$. Its block still has the lower-left minus sign. Do not combine its returned coefficients with an assumed nonnegative $r$.^[ref]

On the graph of a function $z=f(x,y)$, assume $f$ is differentiable, its planar gradient $p$ is nonzero, and speed $U>0$ is measured by the three-dimensional Euclidean norm. The upward unit normal vector is $n=(-p,1)/\sqrt{1+|p{|}^2}$. Projecting the downward vertical vector $-e_z$ onto the tangent plane gives $-(p,|p{|}^2)/(1+|p{|}^2)$. By the Cauchy-Schwarz inequality, normalizing this projected vector produces the tangent direction with greatest height decrease at the prescribed speed, giving the displayed velocity. Its vertical component is $-U|p|/\sqrt{1+|p{|}^2}$, and its horizontal direction is parallel to $-\nabla f$. Consequently its planar paths agree with negative gradient flow up to a change of time, although the speeds differ. Where $p=0$, every tangent direction has zero first-order height change and this normalized formula is undefined.

For an unbiased stochastic gradient $H$ at $X$ with finite conditional second moment, expand $H=\nabla f(X)+(H-\nabla f(X))$ in the squared Euclidean norm. The cross term has zero conditional expectation, which proves the displayed identity. It separates the squared true gradient from the total conditional variance of its random estimate.

Target: gradient-descent.bigb, under Stochastic gradient descent.

A centered random vector is sub-Gaussian with variance proxy ${\tau }^2$ when the displayed inequality holds for every deterministic vector $a$. Equivalently every scalar inner product is a sub-Gaussian random variable with proxy ${\tau }^2\Vert a{\Vert }_2^2$. Independence of coordinates is not required. For vectors $a_j$ with Euclidean norms at most one, the union bound gives ${\max }_j|⟨a_j,g⟩|\le \tau \sqrt{2\log (2m/\delta )}$ with probability at least $1-\delta$. This assumption alone is weaker than concentration for all Lipschitz functions.

A sum of squares adds the squares of numbers $a_i^2$ for specified real numbers $a_i$. It is nonnegative and vanishes exactly when every term is zero. For the coordinates of a vector, it is the squared Euclidean norm; for regression residuals, it gives the residual sum of squares minimized by ordinary least squares.