Numerical linear algebra

Numerical linear algebra develops stable finite algorithms for matrix computations.

A Krylov subspace method seeks approximations in spaces generated by repeated matrix applications to a starting vector. For a linear system with starting point $x_0$ and residual $r_0=b-A{x}_0$, the affine search space is $x_0+K_k(A,r_0)$. Choosing an optimality or orthogonality condition within it gives different algorithms. The conjugate gradient method minimizes the energy norm of the error for a symmetric positive-definite matrix; other methods use residual minimization or different projections. The growth of the Krylov subspace can stop early when an invariant subspace is reached.

An algorithm for solving a linear system $Ax=b$. Tolerance and conditioning can limit a differential-equation calculation even when its time-stepping method is stable.

Matrix preconditioning transforms a linear system using an inexpensive auxiliary solve to improve its eigenvalues. For symmetric positive-definite matrices, a symmetric positive-definite matrix $M$ produces $M^{-1/2}A{M}^{-1/2}$, preserving the symmetric structure needed by the conjugate gradient method.

A useful matrix preconditioner combines a cheap solve with favorable transformed eigenvalues; neither property alone is enough.^[ref]

The aim is to make each correction more effective without making it expensive. The example under diagonal scaling for a two-dimensional linear system turns thin ellipses into nearly circular ones and makes preconditioned steepest descent effective.

A matrix preconditioner is an auxiliary invertible matrix whose matrix inverse is cheap to apply and whose use improves the eigenvalues of a linear system. In symmetric positive-definite matrix problems, a symmetric positive-definite matrix $M$ preserves that structure under the symmetric transformation $M^{-1/2}A{M}^{-1/2}$.

Applying $M^{-1}$ means solving $Mz=r$ for a supplied vector $r$. This notation does not require forming a general matrix inverse. A diagonal matrix gives componentwise divisions; a factored matrix gives auxiliary solves such as forward substitution and back substitution.^[ref]

Choosing $M=I$ makes the solve trivial but changes nothing; choosing $M=A$ gives the exact correction but asks us to solve the original linear system.^[ref]

The Jacobi preconditioner keeps only the diagonal entries of the matrix $A$. For a symmetric positive-definite matrix, these entries are positive, so its auxiliary solve is $z_i=r_i/a_{ii}$. It rescales coordinates with very different diagonal sizes, but cannot remove arbitrary off-diagonal coupling.^[ref]

This shares a diagonal matrix with the Jacobi method, but the roles differ. The Jacobi method specifies a whole stationary iterative method for a linear system; a Jacobi preconditioner supplies an auxiliary solve to another iterative method, including the Preconditioned conjugate gradient method.

Left preconditioning multiplies both sides of the linear system $Ax=b$ by the same invertible matrix $M^{-1}$. It leaves the unknown vector $x$ unchanged. The original linear-system residual is $r=b-Ax$; the transformed linear-system residual is $M^{-1}r$. Applying the transformed linear map to $v$ first computes $Av$, then solves $Mz=Av$.

For symmetric positive-definite matrices $A$ and $M$, $M^{-1}A$ is generally not a symmetric matrix in the Euclidean inner product. Its positive eigenvalues alone do not justify ordinary Euclidean conjugate gradient method recurrences. The remedy is symmetric positive-definite preconditioning, implemented through the Preconditioned conjugate gradient method.

Right preconditioning substitutes $x=M^{-1}u$ into the linear system $Ax=b$. Applying its transformed linear map first solves $Mz=v$, then computes $Az$. Recover the solution by solving $Mx=u$. Its linear-system residual $b-A{M}^{-1}u$ is exactly the original linear-system residual $b-Ax$, which is useful when monitoring the imbalance in the original equations.

The left preconditioning and right preconditioning transformed matrices are similar matrices:They therefore have the same eigenvalues, although their Euclidean norm condition numbers and residual histories need not agree. A similarity transformation preserves eigenvalues; a nonorthogonal change of coordinates need not preserve lengths.

Let $A$ and $M$ be real symmetric positive-definite matrices. The principal square root of a positive semidefinite matrix construction gives an invertible matrix for each. SetThen $Ax=b$ is equivalent to $By=c$. This is a matrix congruence, and $B$ is a symmetric positive-definite matrix: $B^T=B$, and for every nonzero vector $v$,since $S^{-1}v\ne 0$. This short proof establishes both requirements for applying the conjugate gradient method to the transformed linear system.

The left transformed matrix has the same eigenvalues becauseIts correct geometry uses the weighted inner product $⟨u,v{⟩}_M=u^{T}Mv$: $⟨u,M^{-1}Av{⟩}_M=u^{T}Av=⟨M^{-1}Au,v{⟩}_M$. Thus it is symmetric in this inner product, even when it is not a symmetric matrix in ordinary coordinates. The spectral condition number governing the transformed symmetric problem is ${\kappa }_2(B)$, not generally ${\kappa }_2(M^{-1}A)$.

The Preconditioned conjugate gradient method realizes this transformation using auxiliary solves $M{z}_k=r_k$, without constructing $S$ or its matrix inverse. With fixed $M$, initialize $r_0=b-A{x}_0$ and $p_0=z_0$; while $r_k\ne 0$, usethen solve $M{z}_{k+1}=r_{k+1}$ and setStop at a zero linear-system residual before forming another ratio. Positive definiteness makes the denominators positive before termination.^[ref]

An invertible matrix alone is insufficient: with $A=I$ and $M=\mathrm{diag}(-1,1)$, the left transformed matrix has eigenvalues $-1,1$, and $M$ has no real symmetric positive-definite matrix square root. Even a valid symmetric positive-definite matrix can worsen conditioning if its scaling is unsuitable, as the next worked example shows.

For a symmetric positive-definite matrix $A$, solving $A{x}_{\ast }=b$ is equivalent to minimizing the quadratic form with linear termWriting the error vector $e=x-x_{\ast }$ and expanding gives the exact energy norm identityThe cross term vanishes because $A{x}_{\ast }=b$. With $\eta =Se$ and $B=S^{-1}A{S}^{-1}$, the same quantity is ${\eta }^{T}B\eta /2$. Symmetric positive-definite preconditioning changes coordinates without changing the energy assigned to a corresponding error vector.

In an orthonormal eigenbasis of $A$, the level set $e^{T}Ae=t>0$ has semiaxis lengths $\sqrt{t/{\lambda }_i}$. Its longest-to-shortest axis ratio is $\sqrt{{\kappa }_2(A)}$. A large spectral condition number of a positive-definite matrix therefore appears geometrically as a thin ellipsoid. Good scaling makes the transformed ellipsoid rounder, allowing a simple search direction to improve all coordinates together.

The negative gradient of $F$ is the linear-system residual $r$. Relative to the norm $\Vert v{\Vert }_M=\sqrt{v^{T}Mv}$, a steepest direction is $z=M^{-1}r$: the Cauchy-Schwarz inequality applied to $r^{T}v=(S^{-1}r{)}^T(Sv)$ shows that the greatest decrease per unit $M$-length occurs along this direction. Equivalently, ordinary gradient descent in $y=Sx$ becomes preconditioned steepest descent in $x$.^[ref]

Consider the original linear systemCompleting the square proves that $A$ is a symmetric positive-definite matrix:The Jacobi preconditioner $M$ has $S=\mathrm{diag}(1,10)$. Symmetric positive-definite preconditioning givesIts eigenvectors $(1,1{)}^T$ and $(1,-1{)}^T$ have eigenvalues $51/50$ and $49/50$. HenceThe characteristic polynomial of $A$ is ${\lambda }^2-101\lambda +2499/25$. Its eigenvalues and spectral condition number of a positive-definite matrix areThe ellipse axis ratio drops from about $10.0020$ to $1.0202$. The transformation multiplies the second coordinate by ten, bringing the two energy scales together.

Both panels show the same energy level sets, expressed in error coordinates $e=x-x_{\ast }$ and $\eta =M^{1/2}e$. Their coordinate axes have equal scales. The marked displacement $(1,0.1{)}^T$ becomes $(1,1{)}^T$; its energy remains $e^{T}Ae/2=1.02$. These are exact contours of the specified quadratic form.

Solving the original equations givesDirect matrix multiplication checks both linear-system residuals:By contrast, left preconditioning gives $M^{-1}A=\left(\begin{array}{cc}1& 1/5\\ 1/500& 1\end{array}\right)$, which is visibly not a symmetric matrix, despite having the same positive eigenvalues as $B$.

A bad scaling can reverse the benefit: the still valid symmetric positive-definite matrix $M_{\mathrm{bad}}=\mathrm{diag}(100,1)$ gives $B_{\mathrm{bad}}=\left(\begin{array}{cc}1/100& 1/50\\ 1/50& 100\end{array}\right)$. Its Rayleigh quotients on the coordinate unit vectors are $0.01$ and $100$, so ${\kappa }_2(B_{\mathrm{bad}})\ge 10000$. Cheapness and positive definiteness do not guarantee improved conditioning.

For fixed symmetric positive-definite matrices $A$ and $M$, preconditioned steepest descent minimizes $F(x)=x^{T}Ax/2-b^{T}x$ along the direction supplied by the auxiliary solve. Given $x_k$, form the linear-system residual $r_k=b-A{x}_k$. If $r_k=0$, stop: $x_k=x_{\ast }$. Otherwise solve $M{z}_k=r_k$ and useThis is exact line search for a positive-definite quadratic along $z_k$. Indeed,whose derivative vanishes at the displayed positive step size. Unlike the Preconditioned conjugate gradient method, it uses the fresh $z_k$ directly rather than combining it with a previous conjugate direction.

For diagonal scaling for a two-dimensional linear system, start at $x_0=0$. Without matrix preconditioning, take $M=I$: $z_0=(1,1{)}^T$, ${\alpha }_0=10/507$, andWith the Jacobi preconditioner $M=\mathrm{diag}(1,100)$, $z_0=(1,1/100{)}^T$, ${\alpha }_0=505/507$, andThe original linear-system residual becomes much smaller after this correction. Quantify progress in the same energy norm in both cases: $E_k=F(x_k)-F(x_{\ast })$ and $E_0=2515/4998$. Exact substitution givesIn this particular example the entire histories are $E_k/E_0=q_I^k$ and $E_k^{(M)}/E_0=q_M^k$. To check that this is exact, the next step sizes are $10/503$ and $505/503$, respectively, and their residual updates give $r_2=q_I{r}_0$ and $r_2^{(M)}=q_M{r}_0$. Multiplying a linear-system residual by a nonzero scalar leaves the next step size unchanged; every pair of steps repeats the same directions at a smaller scale. The two first-step energy ratios then establish these formulas for all $k$.

The curves come from exact rational updates of the specified linear system, converted to decimal values for plotting. The left panel counts iterations. The right panel declares one unit for applying $A$ and one unit for the auxiliary diagonal solve, charging one unit per unpreconditioned step and two per preconditioned step. It excludes setup and vector operations, so it compares a declared cost model rather than measured runtime. Scaling is advantageous even with that extra charge.

These curves compare steepest descent method iterations. On this two-dimensional linear system, both the conjugate gradient method and the Preconditioned conjugate gradient method terminate within two steps in exact arithmetic. The plot therefore establishes no CG iteration-count advantage for this example; the benefit of scaling for larger problems depends on their eigenvalues and auxiliary-solve costs.

Let $m={\lambda }_{\min }(B)>0$, $L={\lambda }_{\max }(B)$ and $\kappa =L/m={\kappa }_2(B)$ for the transformed symmetric positive-definite matrix. Then preconditioned steepest descent satisfiesThis is a worst-case bound, not necessarily the actual rate. For our scaled example the bound is $(1/50{)}^2=0.0004$, slightly above its computed factor $q_M\approx 0.000384321291$. If the initial transformed error is an eigenvector, exact line search for a positive-definite quadratic removes it in one step, regardless of the full condition number.

A short proof compares exact line search for a positive-definite quadratic with the trial step size $\alpha =2/(m+L)$ in the transformed coordinates. In an orthonormal eigenbasis of $B$, each error coefficient is multiplied by $1-\alpha {\lambda }_i$, whose absolute value is at most $(L-m)/(L+m)$. Summing ${\lambda }_i$ times the squared coefficients bounds the trial energy by the square of this factor. The exact minimizing step size does at least as well. This also explains the terminating case $m=L$: the trial step removes the entire error in one iteration.

Power iteration finds a dominant eigenvalue and its eigenvector by repeatedly applying a matrix and rescaling the resulting vector. The essential idea is selective amplification: components along different eigenvectors grow at different rates, so one direction can eventually dominate. It is the computational form of the power method and is useful when applying a large sparse matrix to a vector is cheap but computing its whole spectral decomposition is expensive.

For a square matrix over the real numbers or complex numbers $A$, choose a unit vector $x_0$ and repeatHere $\Vert \cdot {\Vert }_2$ is the Euclidean norm. If $y_{k+1}=0$, stop before division: the current vector is an eigenvector for the zero eigenvalue, and this iteration cannot continue. Otherwise, in exact arithmetic,Repeated normalization keeps the stored vectors at manageable size without changing their directions; it does not guarantee that an individual multiplication cannot exceed the range of floating-point arithmetic. The dominant eigenvalue means largest modulus, which need not mean largest signed real value.^[ref]

Consider the symmetric matrix and initial unit vectorIts orthonormal eigenbasis consists of $u_1=(1,1{)}^T/\sqrt{2}$ and $u_2=(1,-1{)}^T/\sqrt{2}$, with eigenvalues $3$ and $1$. Since $x_0=(u_1+u_2)/\sqrt{2}$, direct multiplication givesThe first unnormalized vectors are $(1,0{)}^T$, $(2,1{)}^T$, $(5,4{)}^T$, and $(14,13{)}^T$. If ${\theta }_k$ is the acute angle from the line spanned by the dominant eigenvector, then $\tan {\theta }_k=3^{-k}$. Every application divides the unwanted-to-wanted component ratio by three: normalization changes lengths, while the spectral gap selects the direction.

The right panel compares the power iteration direction error, Rayleigh quotient error, and eigenpair residual. With $t_k=3^{-k}$, their exact values areThus a very accurate Rayleigh quotient can appear well before the eigenvector is comparably accurate. This is an example of quadratic Rayleigh-quotient improvement for the power method.

Suppose $A\in {\mathbb{C}}^{n\times n}$ is a diagonalizable matrix, $n\ge 2$, with an eigenbasis $v_1,\dots ,v_n$ and spectral decompositionAssume the dominant eigenvalue ${\lambda }_1$ is a simple eigenvalue andThen the normalized power iteration approaches the one-dimensional eigenspace of ${\lambda }_1$. Its asymptotic direction-error bound has factor $\gamma =|{\lambda }_2/{\lambda }_1|<1$. This spectral gap is a gap in modulus: if $g=|{\lambda }_1|-|{\lambda }_2|$, then $\gamma =1-g/|{\lambda }_1|$. A small relative gap gives slow convergence.^[ref]

The short proof is the exact eigenbasis identityThe remaining sum has norm bounded by a constant times ${\gamma }^k$. This proves convergence of direction, allowing a changing sign or complex factor. A tiny $c_1$ or a poorly conditioned, nonorthogonal eigenbasis can delay visible convergence: the condition number of $V$ affects constants, while $\gamma$ controls the eventual rate. The bound need not be an equality; if the initial coefficient of the second-largest eigenvalue vanishes, the active spectrum of the power method gives a faster rate. When $\gamma =0$, all unwanted components disappear after one multiplication.^[ref]

The hypothesis is $c_1=(V^{-1}{x}_0{)}_1\ne 0$, not an ordinary nonzero inner product with the right eigenvector. Let $w_1^{\ast }$ be the first row of $V^{-1}$, a left eigenvector normalized so $w_1^{\ast }{v}_j$ is $1$ for $j=1$ and $0$ otherwise; $\ast$ denotes conjugate transpose. Then $c_1=w_1^{\ast }{x}_0$. For a Hermitian matrix, choose an orthonormal eigenbasis and the criterion becomes $v_1^{\ast }{x}_0\ne 0$: the starting vector must be nonorthogonal to the dominant eigenvector. For a general diagonalizable matrix, the nonorthogonal eigenbasis makes these two tests different, as illustrated under power iteration failure cases.

For a unit vector $u$ spanning the target eigenspace and a normalized iterate $x_k$, defineThis is the norm of the component in the orthogonal complement of $u$, equivalently the sine of the acute angle to its line. It ignores the arbitrary sign or complex unit factor of an eigenvector. Under power iteration convergence for a diagonalizable matrix, ${\epsilon }_k=O({\gamma }^k)$ in Big O notation. The orthogonal projection defines how we measure direction; it does not assume that the eigenbasis of $A$ is orthogonal. In computation, the projection form avoids subtracting nearly equal numbers inside the square root.

Under power iteration convergence for a diagonalizable matrix, take $v_1=u_1$ to be a unit vector. The displayed formula describes both the dominant eigenvector direction and its rotating complex unit factor. With real data and a positive dominant eigenvalue, $x_k$ converges to $u_1$ or $-u_1$ according to the sign of $c_1$. With a negative dominant eigenvalue, consecutive vectors approach opposite signs even though their power iteration direction error tends to zero.

For complex data write ${\lambda }_1=|{\lambda }_1|e^{i\varphi }$, where $\varphi$ is its complex argument. The leading factor turns by $e^{i\varphi }$ per step, so the vectors generally do not converge. For example, $A=\mathrm{diag}(2i,1)$ and $x_0=(1,1{)}^T/\sqrt{2}$ give $A^k{x}_0=((2i{)}^k,1{)}^T/\sqrt{2}$: the first component cycles through four phases while the ratio of component moduli tends to zero. Its power iteration direction error is $2^{-k}/\sqrt{1+2^{-2k}}$. Multiplying $x_k$ by $e^{-i(\arg c_1+k\varphi )}$ gives a sequence converging to $u_1$.

A stopping rule based only on $\Vert x_{k+1}-x_k{\Vert }_2$ therefore rejects successful sign-alternating iterations. A comparison invariant under complex unit factors isEven this comparison can be small for a slowly changing wrong mixture; the eigenpair residual supplies a separate diagnostic.

For each unit vector iterate compute the Rayleigh quotient ${\rho }_k=x_k^{\ast }A{x}_k$ and the eigenpair residual $r_k=A{x}_k-{\rho }_k{x}_k$. Use the same multiplication $A{x}_k$ for both diagnostics and the next normalized iterate. For any complex matrix, this Rayleigh quotient minimizes $\Vert A{x}_k-\mu x_k{\Vert }_2$ over complex scalars $\mu$, because $x_k^{\ast }{r}_k=0$. Under power iteration convergence for a diagonalizable matrix, ${\rho }_k\to {\lambda }_1$ and $\Vert r_k{\Vert }_2\to 0$, even when the vectors alternate signs or phases.^[ref]

A dimensionless stopping quantity iswhen the denominator is positive; $\Vert A{\Vert }_2$ is the operator norm, and a known upper bound may replace it if computing it is costly. A small eigenpair residual certifies a small backward error of an approximate eigenpair, but does not establish that the eigenvalue found is dominant.

For a Hermitian matrix with orthonormal eigenbasis $u_j$, write $x_k={\sum }_j{a}_j{u}_j$. Then the independently useful identityshows that some eigenvalue lies within $\Vert r_k{\Vert }_2$ of ${\rho }_k$. If a specific target eigenvalue ${\lambda }_1$ is separated from all the others by ${\delta }_k={\min }_{j>1}|{\lambda }_j-{\rho }_k|>0$, it also gives ${\epsilon }_k\le \Vert r_k{\Vert }_2/{\delta }_k$. This separation is measured from the computed Rayleigh quotient; it differs from the dominant-modulus spectral gap. For a nonnormal matrix, the same orthogonal expansion is unavailable and a small eigenpair residual alone need not imply a comparably small distance to the spectrum.

A tied largest modulus can prevent even direction convergence. With $D=\mathrm{diag}(2,-2)$ and $x_0=(1,1{)}^T/\sqrt{2}$,The two directions form a cycle; neither is an eigenvector. There is no dominant-modulus spectral gap. A tied modulus is not a universal failure: an initial eigenvector remains on its line, and $A=2I$ preserves every direction. Subspace iteration addresses a separated cluster by targeting a dominant invariant subspace instead of one eigenvector.

An orthogonal start can fail despite a strict spectral gap. For $A=\left(\begin{array}{cc}2& 1\\ 1& 2\end{array}\right)$ and $x_0=(1,-1{)}^T/\sqrt{2}$, the dominant component is zero, so $A{x}_0=x_0$ and every iterate equals $x_0$. The Rayleigh quotient is exactly $1$ and the eigenpair residual is zero, although the dominant eigenvalue is $3$. Exact arithmetic never creates a missing component.^[ref]

The third panel contrasts these failures with $N=\mathrm{diag}(-3,1)$, $x_0=(1,1{)}^T/\sqrt{2}$, for whichHere the horizontal eigenvector direction converges with a strict spectral gap, while the normalized vectors alternate signs. This is the successful direction convergence described under sign and phase in power iteration.

For a nonorthogonal eigenbasis, even the word “orthogonal” must be used carefully. The diagonalizable matrix $C=\left(\begin{array}{cc}3& 2\\ 0& 1\end{array}\right)$ has right eigenvectors $v_1=(1,0{)}^T$, $v_2=(-1,1{)}^T$, and dominant left eigenvector $w_1=(1,1{)}^T$. For $x_0=(a,b{)}^T$, the dominant coefficient is $c_1=w_1^T{x}_0=a+b$. The starting vector $v_2$ is nonorthogonal to $v_1$ but stays at the smaller eigenvalue $1$; $(0,1{)}^T$ is orthogonal to $v_1$ but has $c_1=1$ and converges to the dominant direction. The left eigenvector test, not the right-eigenvector inner product, controls the outcome.

A random vector drawn from a continuous probability distribution with a probability density function on the full real or complex vector space has $c_1\ne 0$ with probability one. This avoids the exact missing-component hyperplane, but does not guarantee a large component or fast convergence. Floating-point arithmetic may seed a missing component, yet a structurally preserved zero can remain zero, so accidental perturbations are not a convergence guarantee.

Gaussian elimination applies elementary row operations to reduce a linear system to triangular form. Dense elimination uses $O(n^3)$ arithmetic operations.

Gaussian elimination solves a linear system by removing one unknown from later equations at each step. Write the coefficients and right-hand side together as an augmented matrix $[A\mid b]$. Apply the same elementary row operations to both parts until the coefficients form a row echelon form; for a square invertible matrix, this gives an upper triangular matrix system. Then back substitution recovers the unknowns from the last equation upward. The useful simplification is that each successive equation involves fewer unknowns, while the set of solutions stays the same.

Why do elementary row operations preserve solutions? Interchanging equations changes only their order. Multiplying an equation by a nonzero scalar can be undone by division. Replacing equation $i$ by itself plus $c$ times equation $j$, with $i\ne j$, can be undone by adding $-c$ times the unchanged equation $j$. Equivalently, an elementary matrix $E$ is an invertible matrix, so $Ax=b$ if and only if $EAx=Eb$. Applying an operation to $A$ without also applying it to $b$ loses this equivalence.

At step $k$, choose a nonzero pivot element $a_{kk}$ after any row interchange. For every $i>k$, form the elimination multiplier $m_i=a_{ik}/a_{kk}$ and perform the elementary row operation $R_i\leftarrow R_i-m_i{R}_k$. Its coefficient in column $k$ becomes $a_{ik}-(a_{ik}/a_{kk})a_{kk}=0$ in exact arithmetic. Later operations combine rows that already have zeros in earlier columns, so those zeros survive. If an active column has no nonzero candidate, general row echelon form reduction skips it; a square invertible matrix always has a candidate at each step with suitable row interchanges. Partial pivoting also considers roundoff error when selecting the candidate.^[ref]

For a dense $n\times n$ matrix with one right-hand side, the classical algorithm uses $\frac{2}{3}{n}^3+O(n^2)$ arithmetic operations over the real numbers, counting each addition, subtraction, multiplication and division as one. At step $k$, the coefficient update has $(n-k{)}^2$ entries, each costing one multiplication and one subtraction; the elimination multipliers, right-hand-side updates and back substitution contribute lower-order terms. Searching for partial pivoting adds $O(n^2)$ comparisons, and storing the dense coefficients uses $O(n^2)$ space. These assumptions exclude the different storage and work patterns of sparse Gaussian elimination; the exact loop counts appear in LU factorization and solve costs. Here $O$ denotes Big O notation.

Recording the same steps as an LU decomposition makes Reusing LU factors for multiple right-hand sides possible. This is useful whenever the coefficients stay fixed while the forcing vector changes: the expensive elimination is done once. The choice of pivot element matters even when the linear system is well conditioned, as Tiny-pivot roundoff in Gaussian elimination demonstrates.

For a band matrix, elimination without pivoting retains bounded lower and upper bandwidths in its triangular factors. A tridiagonal system therefore needs only linear work and storage, as in the Thomas algorithm. Pivoting can change the precise fill pattern and must be included in the bandwidth and cost analysis when required. The inverse need not be banded, so solving through the factors preserves sparsity that explicit inversion would lose.

A fraction-free elimination algorithm for exact determinants. At each stage it divides a cross-multiplied update by the preceding pivot; the determinantal identities underlying the method make this division exact when its pivot assumptions hold. Pivoting handles zero pivots.

Consider this original linear system and its augmented matrix:The first diagonal entry is zero, so it cannot serve as a pivot element. Interchange $R_1$ and $R_2$ to obtain a pivot element of one:This chooses a convenient nonzero entry for exact arithmetic; partial pivoting would instead interchange $R_1$ and $R_3$, selecting the largest absolute value, two. For our chosen ordering, the first-column elimination multipliers are $m_2=0$ and $m_3=2$. The elementary row operation $R_3\leftarrow R_3-2R_1$ givesThe next pivot element is two and the next elimination multiplier is $7/2$. Applying $R_3\leftarrow R_3-\frac{7}{2}{R}_2$ gives the upper triangular matrix systemEvery operation also changed the right-hand side. Back substitution now givesChecking the original equations, rather than only the reduced ones, verifies the answer by matrix multiplication:Thus a zero first diagonal entry was a row-order obstruction, not a singular matrix. Indeed, the diagonal product of the final upper triangular matrix is $7$; the single row interchange makes $\det A=-7$, since the two row additions preserve the determinant.

A pivot element is the current diagonal entry used to eliminate entries below it in Gaussian elimination, after any row interchange. At step $k$ it is $a_{kk}^{(k-1)}$, where $A^{(k-1)}$ is the current reduced matrix. Computing elimination multipliers requires this entry to be nonzero. A zero pivot element can reflect an unsuitable row order even for an invertible matrix, as Pivoted LU decomposition illustrates.

For $i>k$, an elimination multiplier is the scalar $l_{ik}=a_{ik}^{(k-1)}/a_{kk}^{(k-1)}$ used in the elementary row operation $R_i\leftarrow R_i-l_{ik}{R}_k$. This creates a zero below the pivot element. The remaining entries change by $a_{ij}\leftarrow a_{ij}-l_{ik}{a}_{kj}$ for $j>k$. Storing these elimination multipliers gives the subdiagonal entries of $L$ in LU decomposition; later row interchanges must also reorder the already stored coefficients.

Sparse Gaussian elimination solves a linear system by Gaussian elimination while exploiting its sparse matrix structure to avoid storing zeros and performing unnecessary arithmetic operations. The catch is that zeros in the input need not remain zero: eliminating a shared unknown can couple two equations that previously had no direct coupling. These new positions are fill-in, and their number can depend strongly on the order of elimination.

For a real symmetric positive-definite matrix, the usual direct method is a Cholesky decomposition. Choose an ordering, use symbolic factorization to allocate the predicted factor entries, compute their values, then solve two triangular systems. Compressed sparse storage makes the reduced storage useful. General LU decomposition may also need numerical partial pivoting, so an ordering chosen for sparsity can change during factorization.^[ref]

For a fixed ordering, let $P$ be the permutation matrix that puts the unknowns in that order. Permute both rows and columns:Here $C$ is the lower triangular matrix in the Cholesky decomposition of $B$; $z=Px$. Forward substitution and back substitution give the original solution. The simultaneous permutation preserves the symmetric positive-definite matrix property because $v^{T}Bv=(P^{T}v{)}^{T}A(P^{T}v)>0$ for every nonzero vector $v$. A row permutation alone generally loses the symmetric matrix structure.

The local mechanism is a Schur complement. If the active pivot element is $a_{kk}$, then for remaining indices $i,j$,Even when $a_{ij}=0$, two nonzero couplings through $k$ make the second term nonzero. Thus the elimination graph removes $k$ and joins its remaining neighbors into a clique. The fill-path criterion for symmetric elimination extends this local rule across several steps. The elimination tree organizes the resulting dependencies; the minimum degree algorithm and nested dissection choose orderings to control them.

This connection between graphs and linear systems was already explicit in Donald J. Rose, R. Endre Tarjan and George S. Lueker's 1976 study of fill-in and perfect elimination orderings.^[ref]

Take a star graph $G$ with center $c$ and four leaves $1,2,3,4$. Let $L_G$ be its Laplacian matrix and $I$ the identity matrix. In the order $(c,1,2,3,4)$, the original example isFor a nonzero vector $u$, its quadratic form isThus $A$ is a symmetric positive-definite matrix and either ordering below admits a Cholesky decomposition. Changing the ordering changes the work, rather than the mathematical solvability.

Center first. Eliminating $c$ with pivot element $5$ leaves the Schur complementEvery pair of leaves now has coupling $-1/5$: six new edges, hence six undirected fill-in positions. The four remaining vertices form a clique. In the Cholesky decomposition, the first column has $C_{cc}=\sqrt{5}$ and $C_{ic}=-1/\sqrt{5}$. The rest of the lower triangular matrix is dense. To check that later cancellation does not remove its entries, after $r$ leaves have also been eliminated, the active leaf block isIndeed a leaf step sends $\alpha$ to $\alpha +{\alpha }^2/(2-\alpha )$. All active off-diagonal entries remain negative. The successive pivot elements in ordinary symmetric Gaussian elimination, equivalently the squared diagonals in the Cholesky decomposition, areThe Cholesky decomposition therefore stores $5+4+3+2+1=15$ nonzero entries, including its diagonal.