= Singular value decomposition
{title2=$A=U\Sigma V^*$}
{wiki}
= SVD
{c}
{synonym}
= Biunitary diagonalization
{synonym}
The <singular value decomposition> separates any <matrix> into input directions, stretching factors, and output directions. Every $m\times n$ complex <matrix> admits
$$
A=U\Sigma V^*,\qquad U\in\mathbb C^{m\times m},\quad
\Sigma\in\mathbb R^{m\times n},\quad V\in\mathbb C^{n\times n}.
$$
Here $U,V$ are <unitary matrices>, $V^*=V^\dagger$ is the <conjugate transpose>, and $\Sigma$ is a rectangular <diagonal matrix>. Its $p=\min(m,n)$ diagonal entries are the <singular values>, ordered $\sigma_1\geq\cdots\geq\sigma_p\geq0$. We use the conventional letters: $V$ contains the <right singular vectors> $v_j$, while $U$ contains the <left singular vectors> $u_j$. Writing $A=V\Sigma U^\dagger$ instead merely swaps these two matrix names; it requires swapping their roles consistently. For real <matrices>, choose <orthogonal matrices> and replace the <conjugate transpose> with the <transpose>.https://math.uchicago.edu/~may/REU2016/REUPapers/Mazeika.pdf[Mazeika, The singular value decomposition and low rank approximation, §3]{ref}
Reading from right to left, $V^*$ expresses the input in an <orthonormal basis>, $\Sigma$ stretches or annihilates those coordinates, and $U$ assembles the output in another <orthonormal basis>. Unlike <matrix similarity>, the input and output <bases> can differ, and even have different sizes. A $3\times2$ <matrix> has full factors of sizes $3\times3$, $3\times2$, and $2\times2$. If its <matrix rank> is $2$, one can retain only the active columns, giving sizes $3\times2$, $2\times2$, and $2\times2$ for $U_r,\Sigma_r,V_r^*$; the full $U$ additionally describes an unused output direction.
For a fully computed nonsymmetric example, take
$$
A=\begin{pmatrix}3&4\\-4/5&3/5\end{pmatrix},\qquad
U=I,\quad\Sigma=\begin{pmatrix}5&0\\0&1\end{pmatrix},\quad
V=\frac15\begin{pmatrix}3&-4\\4&3\end{pmatrix}.
$$
Multiplication gives $U\Sigma V^T=A$. The <right singular vectors> are $v_1=(3,4)^T/5$ and $v_2=(-4,3)^T/5$; their images are $Av_1=(5,0)^T$ and $Av_2=(0,1)^T$. Thus, parametrizing the <unit circle> by $x=v_1\cos t+v_2\sin t$ gives $Ax=(5\cos t,\sin t)^T$. Its image is the <ellipse> $y_1^2/25+y_2^2=1$. The two <singular values> are its semiaxis lengths, and the <left singular vectors> point along its axes. The <singular value decomposition> finds precisely the perpendicular input directions whose images stay perpendicular.
\Image[/singular-value-ellipse.png]
{title=The <unit circle> becomes an <ellipse>: $Av_1=5u_1$ and $Av_2=u_2$}
For compression, write the <matrix> as a sum of <outer products> and retain the largest terms:
$$
A=\sum_{j=1}^r\sigma_j u_jv_j^*,\qquad
A_k=\sum_{j=1}^k\sigma_j u_jv_j^*,\qquad
\|A-A_k\|_F^2=\sum_{j=k+1}^r\sigma_j^2,
$$
where $r$ is the <matrix rank>. Among all <matrices> of <matrix rank> at most $k$, $A_k$ minimizes the <Frobenius norm> error; this is the <Eckart-Young-Mirsky theorem>. The <operator norm> error is $\sigma_{k+1}$ for $k<r$. These statements concern specified <norms>, rather than guaranteeing that discarded details are visually or scientifically unimportant.https://math.uchicago.edu/~may/REU2016/REUPapers/Mazeika.pdf[Mazeika, §4.3: low rank approximation]{ref}
Consider an $8\times10$ brightness <matrix> $B=pq^T+ab^T/2$, with
$$
p=(1,2,3,4,4,3,2,1)^T,\qquad q=(1,2,3,4,5,5,4,3,2,1)^T,
$$
and $a_i=(-1)^{i-1}$, $b_j=(-1)^{j-1}$. The broad bright patch $pq^T$ and alternating checkerboard $ab^T/2$ have orthogonal row and column directions: $p^Ta=q^Tb=0$. Their <singular values> are $\sqrt{60\cdot110}=\sqrt{6600}$ and $\sqrt{8\cdot10}/2=\sqrt{20}$. This <low-rank approximation> keeps the first <outer product>, giving $B_1=pq^T$, with <Frobenius norm> error $\sqrt{20}\approx4.472$ and relative error $\sqrt{20/6620}\approx5.50\%$. For example, $B_{11}=1.5$ becomes $1$, and $B_{12}=1.5$ becomes $2$: every entry changes by $0.5$ in magnitude. The discarded pattern is small but entirely lost.
\Image[/singular-value-rank-one.png]
{title=An $8\times10$ <matrix>, its rank-one approximation, and the discarded checkerboard}
For this illustrative <image compression>, storing one normalized <left singular vector>, one normalized <right singular vector>, and one <singular value> requires $8+10+1=19$ scalars instead of $80$. This counts scalars at equal precision, before storage overhead. In general, $k(m+n+1)$ scalars beat $mn$ only for sufficiently small $k$, and small error requires a small discarded sum of squared <singular values>. The usefulness comes from measuring the tradeoff before throwing information away.https://math.mit.edu/~gs/linearalgebra/ila5/SVD_Slides.pdf[Verma, Singular value decomposition of real matrices, slides 9 and 15–16]{ref}
Back to article page
