Heavy-ball method

The heavy-ball iteration adds momentum to gradient descent:

For $F(x)=x^{T}Ax/2-b^{T}x$ and $r_k=b-A{x}_k$,Thus $r_k$ is a polynomial in $A$ applied to the initial residual and lies in the corresponding Krylov subspace.

For $e_k=x^{\ast }-x_k$,If $A$ is diagonal with entries ${\lambda }_i$, a coordinate permutation turns this matrix into blocks

For $A=\mathrm{diag}(1,\gamma )$, $\alpha =1/\gamma$, and $\beta =(1-{\gamma }^{-1/2}{)}^2$, every error-propagation block has spectral radius at most $1-{\gamma }^{-1/2}$, and equality occurs.