Past exam of the numerical linear algebra course of the University of Oxford / 2011 / 3

For the quadratic function, write $r_j=b-A{x}_j$ and restrict to the full line $x_j+\alpha p_j$. Symmetry of $A$ givesThe coefficient $p_j^{T}A{p}_j$ is positive by positive definiteness and $p_j\ne 0$. The unique minimizer is thereforeQuadratic minimization along a line also gives the exact changeThus strict decrease holds exactly when $p_j^T{r}_j\ne 0$. The search is over the whole line, so ${\alpha }_j$ can be negative; a positive step size would require the additional descent direction condition $p_j^T{r}_j>0$.

The exact minimization gives $r_{j+1}=r_j-{\alpha }_{j}A{p}_j$, with ${\alpha }_j=p_j^T{r}_j/(p_j^{T}A{p}_j)$. For steepest descent method, the next search direction is $p_{j+1}=r_{j+1}$. HenceThus consecutive nonzero search directions are orthogonal. This calculation even covers an arbitrary initial direction $p_1$, as in the generic algorithm: it only needs the exact preceding line minimization and the choice $p_{j+1}=r_{j+1}$. If the new linear-system residual is zero, the linear system is already solved. No pairwise orthogonality of all nonconsecutive directions is asserted.

Nonzero vectors $p_1,\dots ,p_k$ are conjugate directions with respect to $A$ when $p_i^{T}A{p}_j=0$ for $i\ne j$. They are linearly independent: if ${\sum }_i{c}_i{p}_i=0$, multiplication by $p_j^{T}A$ gives $c_j{p}_j^{T}A{p}_j=0$, hence $c_j=0$ by positive definiteness. There are consequently at most $n$ such directions.

The finite termination of conjugate directions follows from the residual orthogonality maintained by the exact line minimizations. At step $j$,For $i<j$, conjugacy gives $p_i^T{r}_{j+1}=p_i^T{r}_j$. By mathematical induction, $r_{j+1}$ is orthogonal to every $p_1,\dots ,p_j$. After $n$ nonzero conjugate directions they form a basis of ${\mathbb{R}}^n$, forcing $r_{n+1}=0$. Thus the method reaches $Ax=b$ in at most $n$ steps, and positive definiteness makes that solution unique. The claim is in exact arithmetic; rounding errors need not preserve exact conjugacy.

Since $A{p}_1=(5,2{)}^T$, an A-conjugate direction $p_2=(a,b{)}^T$ must satisfy $5a+2b=0$. One nonzero choice isDirect matrix multiplication gives $A{p}_2=(3,-3{)}^T$, and $p_1^{T}A{p}_2=0$. All nonzero scalar multiples of this vector also qualify.