Singular value decomposition ($A=U\Sigma V^{\ast }$)

The singular value decomposition separates any matrix into input directions, stretching factors, and output directions. Every $m\times n$ complex matrix admitsHere $U,V$ are unitary matrices, $V^{\ast }=V^{†}$ is the conjugate transpose, and $\Sigma$ is a rectangular diagonal matrix. Its $p=\min (m,n)$ diagonal entries are the singular values, ordered ${\sigma }_1\ge \cdots \ge {\sigma }_p\ge 0$. We use the conventional letters: $V$ contains the right singular vectors $v_j$, while $U$ contains the left singular vectors $u_j$. Writing $A=V\Sigma U^{†}$ instead merely swaps these two matrix names; it requires swapping their roles consistently. For real matrices, choose orthogonal matrices and replace the conjugate transpose with the transpose.^[ref]

Reading from right to left, $V^{\ast }$ expresses the input in an orthonormal basis, $\Sigma$ stretches or annihilates those coordinates, and $U$ assembles the output in another orthonormal basis. Unlike matrix similarity, the input and output bases can differ, and even have different sizes. A $3\times 2$ matrix has full factors of sizes $3\times 3$, $3\times 2$, and $2\times 2$. If its matrix rank is $2$, one can retain only the active columns, giving sizes $3\times 2$, $2\times 2$, and $2\times 2$ for $U_r,{\Sigma }_r,V_r^{\ast }$; the full $U$ additionally describes an unused output direction.

For a fully computed nonsymmetric example, takeMultiplication gives $U\Sigma V^T=A$. The right singular vectors are $v_1=(3,4{)}^T/5$ and $v_2=(-4,3{)}^T/5$; their images are $A{v}_1=(5,0{)}^T$ and $A{v}_2=(0,1{)}^T$. Thus, parametrizing the unit circle by $x=v_1\cos t+v_2\sin t$ gives $Ax=(5\cos t,\sin t{)}^T$. Its image is the ellipse $y_1^2/25+y_2^2=1$. The two singular values are its semiaxis lengths, and the left singular vectors point along its axes. The singular value decomposition finds precisely the perpendicular input directions whose images stay perpendicular.

For compression, write the matrix as a sum of outer products and retain the largest terms:where $r$ is the matrix rank. Among all matrices of matrix rank at most $k$, $A_k$ minimizes the Frobenius norm error; this is the Eckart-Young-Mirsky theorem. The operator norm error is ${\sigma }_{k+1}$ for $k<r$. These statements concern specified norms, rather than guaranteeing that discarded details are visually or scientifically unimportant.^[ref]

Consider an $8\times 10$ brightness matrix $B=p{q}^T+a{b}^T/2$, withand $a_i=(-1{)}^{i-1}$, $b_j=(-1{)}^{j-1}$. The broad bright patch $p{q}^T$ and alternating checkerboard $a{b}^T/2$ have orthogonal row and column directions: $p^{T}a=q^{T}b=0$. Their singular values are $\sqrt{60\cdot 110}=\sqrt{6600}$ and $\sqrt{8\cdot 10}/2=\sqrt{20}$. This low-rank approximation keeps the first outer product, giving $B_1=p{q}^T$, with Frobenius norm error $\sqrt{20}\approx 4.472$ and relative error $\sqrt{20/6620}\approx 5.50\%$. For example, $B_{11}=1.5$ becomes $1$, and $B_{12}=1.5$ becomes $2$: every entry changes by $0.5$ in magnitude. The discarded pattern is small but entirely lost.

For this illustrative image compression, storing one normalized left singular vector, one normalized right singular vector, and one singular value requires $8+10+1=19$ scalars instead of $80$. This counts scalars at equal precision, before storage overhead. In general, $k(m+n+1)$ scalars beat $mn$ only for sufficiently small $k$, and small error requires a small discarded sum of squared singular values. The usefulness comes from measuring the tradeoff before throwing information away.^[ref]

If a matrix with orthonormal columns is divided into two row blocks, each having at least as many rows as there are columns, it admits simultaneous thin factorizations $A_1=U_{1}C{V}^T$, $A_2=U_{2}S{V}^T$, with orthonormal $U_1,U_2$, an orthogonal matrix $V$, and nonnegative diagonal matrices $C,S$ satisfying $C^2+S^2=I$. To prove this, apply the spectral theorem to $A_1^T{A}_1$; the same orthonormal eigenbasis diagonalizes $A_2^T{A}_2=I-A_1^T{A}_1$. Normalize the nonzero columns of $A_{i}V$ and complete the zero columns to orthonormal sets. The entries of $C,S$ can be written as cosines and sines of angles in $[0,\pi /2]$.

Every complex symmetric matrix admits a unitary congruence decomposition with a unitary matrix $U$ and nonnegative diagonal entries $s_j$ equal to its singular values. This differs from unitary similarity diagonalization. It reduces a multimode squeezed vacuum to independent oscillator squeezing factors.

The singular values of a matrix measure stretching of Euclidean norm. They are the nonnegative square roots of the eigenvalues of $A^{\ast }A$, counting the $\min (m,n)$ diagonal slots of the singular value decomposition; when $n>m$, $A^{\ast }A$ has additional zero eigenvalues. Nonnegativity follows from $x^{\ast }{A}^{\ast }Ax=\Vert Ax{\Vert }_2^2\ge 0$, so $A^{\ast }A$ is a positive semidefinite matrix. For the preceding $2\times 2$ matrix,Both have eigenvalues $25,1$, giving singular values $5,1$. In particular,Here the operator norm uses Euclidean norm: no unit vector stretches more than fivefold. Zero singular values identify lost directions; the number of positive singular values equals the matrix rank. This summarizes both amplification and information loss with nonnegative numbers.^[ref]

Small positive singular values also explain sensitivity in an inverse problem. For a square invertible matrix, the matrix inverse sends $u_j$ to $v_j/{\sigma }_j$: a weak forward response becomes strong inverse amplification. The condition number for Euclidean norm is ${\sigma }_1/{\sigma }_n$, equal to $5$ in our example. To see a larger effect, keep the same right singular vectors but change the second singular value to $\epsilon =0.01$:The output perturbation has Euclidean norm $0.001$, but the recovered input changes by $0.1$. Its hundredfold amplification follows from the direction of the perturbation, not just its size. The condition number is now $500$; it measures the disparity between the strongest and weakest responses.^[ref]

For a real matrix $A$, let $B=A^{\mathsf{T}}A$. It is positive semidefinite, and the spectral theorem for real symmetric matrices gives $x^{\mathsf{T}}Bx\ge {\lambda }_{\min }\Vert x{\Vert }^2$. If ${\lambda }_{\min }>0$, rescaling any longer vector to unit length decreases its image norm, so minimizers lie on the unit sphere. Lagrange multipliers applied to $x^{\mathsf{T}}Bx$ with $\Vert x\Vert =1$ give $Bx=\lambda x$. Thus the minimizers are exactly the unit vectors in the least eigenspace.

If ${\lambda }_{\min }=0$, the minimum is zero and the full argmin consists of all $x\in \ker A$ with $\Vert x\Vert \ge 1$. Indeed, $\ker (A^{\mathsf{T}}A)=\ker A$ because $x^{\mathsf{T}}{A}^{\mathsf{T}}Ax=\Vert Ax{\Vert }^2$. Restricting the minimizers to the sphere would omit longer kernel vectors in this case. For a ball constraint $\Vert x\Vert \le 1$, the argmin is always $\ker A$ intersected with the ball.

For equally sized matrices, corresponding ordered singular values differ by at most the spectral norm of the perturbation. Indeed, let $B_{j-1}$ be a best approximation of matrix rank at most $j-1$. The Eckart-Young-Mirsky theorem and the triangle inequality give ${\sigma }_j(A)\le \Vert A-B_{j-1}{\Vert }_2\le \Vert A-B{\Vert }_2+{\sigma }_j(B)$. Interchanging $A,B$ proves the bound.

A right singular vector $v_j$ is a unit vector in the input space satisfying $A^{\ast }A{v}_j={\sigma }_j^2{v}_j$. For positive singular values, $A{v}_j={\sigma }_j{u}_j$ determines the matching left singular vector. In the ellipse example, $A{v}_1=(5,0{)}^T$ is not parallel to $v_1$: a right singular vector need not be an eigenvector of $A$. Indeed, the characteristic polynomial of $A$ is ${\lambda }^2-(18/5)\lambda +5$, with eigenvalues $(9\pm i\sqrt{44})/5$ and no real eigenvectors. Nevertheless, its real right singular vectors give a complete orthonormal basis. Eigenvectors describe preserved directions; right singular vectors describe orthogonal input channels of stretching.^[ref]

An arbitrary input mixes these channels. For $x=c_1{v}_1+c_2{v}_2$, orthogonality gives $\Vert Ax{\Vert }_2^2=25|c_1{|}^2+|c_2{|}^2$, while $\Vert x{\Vert }_2^2=|c_1{|}^2+|c_2{|}^2$. Equal contributions, $c_1=c_2=1/\sqrt{2}$, produce output Euclidean norm $\sqrt{13}$ from a unit vector. Concentrating the entire input along $v_1$ produces the maximum $5$. This is why the right singular vectors give a direct reading of how any input will behave. For a compact operator, the same definition describes the positive terms of its singular value system.

A left singular vector $u_j$ is a unit vector in the output space satisfying $A{A}^{\ast }{u}_j={\sigma }_j^2{u}_j$. For positive singular values, the pairing also satisfies $A^{\ast }{u}_j={\sigma }_j{v}_j$. Thus $u_1=(1,0{)}^T$ and $u_2=(0,1{)}^T$ describe the output axes of the preceding ellipse, while the right singular vectors describe its input directions. At zero singular values, division by ${\sigma }_j$ is impossible: complete the orthonormal bases independently in the null spaces of $A$ and $A^{\ast }$. Repeated singular values allow different matched orthonormal bases within the corresponding eigenspaces. The same positive pairing extends to a compact operator through its singular value system.

Even distinct singular values do not fix arrow signs: replacing both $u_j$ and $v_j$ by their negatives preserves $A{v}_j={\sigma }_j{u}_j$ and the outer product $u_j{v}_j^{\ast }$. Reversing only one arrow would change that term's sign. The paired directions, rather than independently chosen arrow orientations, encode the matrix.