Krylov subspace (${\mathcal{K}}_k(A,b)$)

The order-$k$ Krylov subspace generated by $A$ and $b$ is

With zero initial guess, the generalized minimal residual method minimizes the Euclidean norm of the linear-system residual over each Krylov subspace. The Arnoldi decomposition reduces this to ${\min }_y\Vert {\tilde{H}}_{k}y-\Vert b{\Vert }_2{e}_1{\Vert }_2$, followed by $x_k=Q_{k}y$. Givens rotations solve the small problem in $O(k^2)$ operations once the decomposition is available. For an invertible matrix $A$, the same iterate minimizes the error in the weighted norm induced by $A^{T}A$, because $\Vert Ax-b{\Vert }_2=\Vert x-x_{\ast }{\Vert }_{A^{T}A}$.

One step of restarted GMRES searches along the current residual. In the displayed nonsingular system, the first residual minimization is ${\min }_{\alpha }[(1-\alpha {)}^2+(1+\alpha {)}^2]$, whose minimizer is zero. Restarting then repeats the same problem forever and never moves. Unrestarted GMRES instead solves exactly at step two, since $b,Ab$ span the entire space. Thus a smaller search-space cost can come at the price of stagnation; nonsingularity alone does not guarantee progress for one-step restarting.

Starting with $q_1=b/\Vert b{\Vert }_2$, the Arnoldi iteration constructs an orthonormal basis of successive Krylov subspaces. At step $j$, orthogonalize $A{q}_j$ against $q_1,\dots ,q_j$, and normalize the remainder as $q_{j+1}$ if it is nonzero. The coefficients form an upper Hessenberg matrix ${\tilde{H}}_k$ and satisfy $A{Q}_k=Q_{k+1}{\tilde{H}}_k$. A zero remainder means the current Krylov subspace is invariant under $A$.

For a real symmetric matrix $A$, the leading block $Q_k^{T}A{Q}_k$ of the Arnoldi decomposition is both symmetric and an upper Hessenberg matrix, hence a tridiagonal matrix. The resulting three-term recurrence is the Lanczos algorithm. If $A$ is a symmetric positive-definite matrix, its compression is too, since $y^T{Q}_k^{T}A{Q}_{k}y=(Q_{k}y{)}^{T}A(Q_{k}y)>0$ for nonzero $y$.

The factorization produced by the Arnoldi iteration, with orthonormal $Q_{k+1}$ and an upper Hessenberg matrix ${\tilde{H}}_k$. Multiplying by $Q_k^T$ identifies its leading square block as $H_k=Q_k^{T}A{Q}_k$.

For a diagonalizable finite-dimensional operator, write $v={\sum }_j{u}_j$ with nonzero $u_j$ in distinct eigenspaces of eigenvalues ${\lambda }_j$. Then $A^{r}v={\sum }_j{\lambda }_j^r{u}_j$. The Vandermonde matrix of the first $k$ powers is invertible for $k$ distinct eigenvalues, so these powers span exactly the $k$ independent components $u_j$. Thus the stabilized Krylov subspace dimension is $k$. It counts distinct spectral components, not the number of nonzero coefficients in an arbitrary eigenbasis: $A=I$, $v=e_1+e_2$ has Krylov dimension one. Equivalently it is the minimum number of eigenvectors needed when those vectors may be chosen freely.