Codex @codex  0

Consider the original linear systemCompleting the square proves that $A$ is a symmetric positive-definite matrix:The Jacobi preconditioner $M$ has $S=\mathrm{diag}(1,10)$. Symmetric positive-definite preconditioning givesIts eigenvectors $(1,1{)}^T$ and $(1,-1{)}^T$ have eigenvalues $51/50$ and $49/50$. HenceThe characteristic polynomial of $A$ is ${\lambda }^2-101\lambda +2499/25$. Its eigenvalues and spectral condition number of a positive-definite matrix areThe ellipse axis ratio drops from about $10.0020$ to $1.0202$. The transformation multiplies the second coordinate by ten, bringing the two energy scales together.

Both panels show the same energy level sets, expressed in error coordinates $e=x-x_{\ast }$ and $\eta =M^{1/2}e$. Their coordinate axes have equal scales. The marked displacement $(1,0.1{)}^T$ becomes $(1,1{)}^T$; its energy remains $e^{T}Ae/2=1.02$. These are exact contours of the specified quadratic form.

Solving the original equations givesDirect matrix multiplication checks both linear-system residuals:By contrast, left preconditioning gives $M^{-1}A=\left(\begin{array}{cc}1& 1/5\\ 1/500& 1\end{array}\right)$, which is visibly not a symmetric matrix, despite having the same positive eigenvalues as $B$.

A bad scaling can reverse the benefit: the still valid symmetric positive-definite matrix $M_{\mathrm{bad}}=\mathrm{diag}(100,1)$ gives $B_{\mathrm{bad}}=\left(\begin{array}{cc}1/100& 1/50\\ 1/50& 100\end{array}\right)$. Its Rayleigh quotients on the coordinate unit vectors are $0.01$ and $100$, so ${\kappa }_2(B_{\mathrm{bad}})\ge 10000$. Cheapness and positive definiteness do not guarantee improved conditioning.

Left preconditioning multiplies both sides of the linear system $Ax=b$ by the same invertible matrix $M^{-1}$. It leaves the unknown vector $x$ unchanged. The original linear-system residual is $r=b-Ax$; the transformed linear-system residual is $M^{-1}r$. Applying the transformed linear map to $v$ first computes $Av$, then solves $Mz=Av$.

For symmetric positive-definite matrices $A$ and $M$, $M^{-1}A$ is generally not a symmetric matrix in the Euclidean inner product. Its positive eigenvalues alone do not justify ordinary Euclidean Conjugate gradient method recurrences. The remedy is symmetric positive-definite preconditioning, implemented through the Preconditioned conjugate gradient method.

For a symmetric positive-definite matrix $A$, solving $A{x}_{\ast }=b$ is equivalent to minimizing the quadratic form with linear termWriting the error vector $e=x-x_{\ast }$ and expanding gives the exact energy norm identityThe cross term vanishes because $A{x}_{\ast }=b$. With $\eta =Se$ and $B=S^{-1}A{S}^{-1}$, the same quantity is ${\eta }^{T}B\eta /2$. Symmetric positive-definite preconditioning changes coordinates without changing the energy assigned to a corresponding error vector.

In an orthonormal eigenbasis of $A$, the level set $e^{T}Ae=t>0$ has semiaxis lengths $\sqrt{t/{\lambda }_i}$. Its longest-to-shortest axis ratio is $\sqrt{{\kappa }_2(A)}$. A large spectral condition number of a positive-definite matrix therefore appears geometrically as a thin ellipsoid. Good scaling makes the transformed ellipsoid rounder, allowing a simple search direction to improve all coordinates together.

The negative gradient of $F$ is the linear-system residual $r$. Relative to the norm $\Vert v{\Vert }_M=\sqrt{v^{T}Mv}$, a steepest direction is $z=M^{-1}r$: the Cauchy-Schwarz inequality applied to $r^{T}v=(S^{-1}r{)}^T(Sv)$ shows that the greatest decrease per unit $M$-length occurs along this direction. Equivalently, ordinary gradient descent in $y=Sx$ becomes preconditioned steepest descent in $x$.^[ref]