Moore-Penrose inverse ($A^{+}$)

The Moore-Penrose inverse extends the useful part of a matrix inverse to rectangular and singular matrices. A linear system $Ax=b$ may have no exact solution, or may have many. The pseudoinverse resolves both ambiguities: $x_{\ast }=A^{+}b$ is the minimum-norm least-squares solution. First minimize the Euclidean norm $\Vert Ax-b{\Vert }_2$; among all vectors attaining that minimum, choose the one with smallest Euclidean norm $\Vert x{\Vert }_2$. It does not pretend that inconsistent equations can all be satisfied.

For $A\in {\mathbb{C}}^{m\times n}$, its Moore-Penrose inverse is the unique $n\times m$ matrix satisfying the four Penrose equations. The singular value decomposition gives a direct construction:Here $U,V$ are unitary matrices, the star is the conjugate transpose, and ${\Sigma }^{+}$ is the $n\times m$ rectangular diagonal matrix obtained by replacing each nonzero singular value ${\sigma }_j$ by $1/{\sigma }_j$ and leaving zeros at zero. Over the real numbers, use orthogonal matrices and transposes. The pseudoinverse reverses the stretches that can be reversed; it discards output components the original linear map cannot produce and does not invent an input component in its kernel. ^[ref]

The two products have especially clear meanings:They are orthogonal projection matrices: $A{A}^{+}$ selects the producible part of an output, while $A^{+}A$ selects the detectable part of an input. They are generally not identity matrices. For an invertible matrix, both are identities and $A^{+}=A^{-1}$.

This gives a useful order of operations for least-squares solutions: project the requested output onto the column space, then invert the restriction of $A$ from $(\ker A{)}^{\perp }$ to its column space. Adding a kernel vector leaves the residual unchanged but increases the squared Euclidean norm by the Pythagorean theorem in an inner-product space, because $A^{+}b$ is orthogonal to that kernel. The pseudoinverse therefore provides one canonical answer without hiding either lost information or irreconcilable data.

The name records a history with several contributors. E. H. Moore gave an earlier generalization in 1920; M. H. Pearl's 1966 paper records that work, Arne Bjerhammar's 1951 rediscovery for maximal-rank rectangular matrices, and Roger Penrose's independent 1955 treatment. ^[ref] Penrose's original paper proves existence and uniqueness from the four Penrose equations for arbitrary complex matrices, including rectangular ones. ^[ref]

Suppose three measurements constrain two unknowns:The linear system asks for $x_1=1$, $x_2=2$, and $x_1+x_2=6$, which are incompatible. A least-squares solution makes a balanced compromise. The matrix has full column rank, soThe fitted output and residual areThus the two direct measurements are each overshot by one, and the measured sum is undershot by one. The residual has Euclidean norm $\sqrt{3}$ and is orthogonal to the column space, the plane $z=y_1+y_2$ in output coordinates $(y_1,y_2,z)$.

For any perturbation $h\in {\mathbb{R}}^2$, the orthogonality of $r$ to $Ah$ givesSince the kernel of $A$ is trivial, every nonzero $h$ increases the residual. The least-squares solution is unique in this example. The formula using $A^{T}A$ is convenient for these small exact calculations; an actual numerical computation can use a QR decomposition or singular value decomposition to avoid the extra conditioning cost of the least-squares normal equations. For full column rank, the squared singular values of $A$ are the eigenvalues of $A^{T}A$, so its condition number in the Euclidean norm is ${\kappa }_2(A^{T}A)={\kappa }_2(A{)}^2$. ^[ref]

Now take one equation for two unknowns:The linear system $x_1+2x_2=3$ has an entire affine line of exact solutions. Since $A$ has full row rank, its Moore-Penrose inverse isThe kernel is $\mathrm{span}\{(-2,1{)}^T\}$, and all exact solutions areBecause $x_{\ast }$ is orthogonal to $(-2,1{)}^T$,The Euclidean norm is therefore smallest exactly at $t=0$. The zero residual alone cannot select a solution; the minimum-norm rule supplies the additional criterion.

This selection depends on the specified inner product. Rescaling one unknown changes what counts as a small Euclidean norm, so units and weighting matter in an application. A minimum-norm answer is mathematically canonical for the chosen geometry, rather than an automatic guarantee of physical plausibility.

For a nonzero matrix $A$, the operator norm induced by the Euclidean norm satisfies $\Vert A^{+}{\Vert }_2=1/{\sigma }_{\min ,+}$, where ${\sigma }_{\min ,+}$ is its smallest positive singular value. The singular value decomposition makes the danger visible: a small nonzero stretch becomes a large amplification of errors when inverted.

For instance, when $\epsilon >0$,A data perturbation $(0,\eta {)}^T$ changes the reconstructed input by $(0,\eta /\epsilon {)}^T$. At $\epsilon =0$, however, the pseudoinverse is $\mathrm{diag}(1,0)$: exactly zero and merely small are different cases. Thus the pseudoinverse need not be continuous across a change of matrix rank. With uncertain data, truncated singular value decomposition or Tikhonov regularization can trade exact inversion of weak directions for stability; the cutoff or penalty adds a modelling choice.

For a bounded operator $A:X\to Y$ between Hilbert spaces, $A^{†}$ has domain $\mathcal{R}(A)\oplus \mathcal{R}(A{)}^{\perp }$, maps into $\mathcal{N}(A{)}^{\perp }$, and inverts the restriction of $A$ to $\mathcal{N}(A{)}^{\perp }$. It is bounded exactly when $\mathcal{R}(A)$ is closed.

The Moore--Penrose inverse is characterized by

Here $A^{†}=A^{+}$ denotes the Moore-Penrose inverse, while the star denotes the conjugate transpose. The first two identities express consistency when applying and reversing the linear map; the last two make the resulting orthogonal projection matrices Hermitian. They must hold together. For example, $A=(12)$ and $B=(10{)}^T$ satisfy $ABA=A$ and $BAB=B$, butis not Hermitian. This $B$ is a right inverse, yet $Bb=(3,0{)}^T$ is not the minimum-norm least-squares solution for $b=3$. The last two Penrose equations impose the orthogonality that selects the pseudoinverse.

The identity $(AB{)}^{†}=B^{†}{A}^{†}$ does not hold for arbitrary matrices. A sufficient condition is that $A$ have full column rank and $B$ have full row rank.