Past exam of the numerical linear algebra and continuous optimisation course of the University of Oxford / 2010 / B / 4

At an unconstrained local minimizer of a twice continuously differentiable objective function, the first-order optimality condition is $\boxed{\nabla f(x_{\ast })=0}$. The second-order necessary condition is $d^T{\nabla }^{2}f(x_{\ast })d\ge 0$ for every direction $d$, so the Hessian matrix is positive semidefinite.

For the Euclidean norm, the unit steepest descent direction is $-g/\Vert g{\Vert }_2$ when $g=\nabla f(x)\ne 0$; the algorithm usually uses the unnormalized direction $p=-g$. It is a descent direction because $\boxed{g^{T}p=-\Vert g{\Vert }_2^2<0}$.

Two limitations of unpreconditioned steepest descent method in high-accuracy smooth optimization are its slow zigzagging on ill-conditioned valleys and its sensitivity to scaling of the coordinates. It uses no curvature information to correct those directions, so it can require many function and gradient evaluations compared with a suitable Newton or quasi-Newton method. These are reasons to prefer curvature-aware methods in that setting, rather than a claim that gradient-based methods are never useful.

The descent lemma follows by integrating the gradient along the line and using its Lipschitz continuity:This upper bound is at most the right side of the Armijo condition when $\alpha [(1-\beta )g^{T}p+\gamma \alpha \Vert p{\Vert }_2^2/2]\le 0$. Since $g^{T}p<0$, it holds forThe zero step gives equality; every positive step in this interval satisfies the prescribed sufficient-decrease test.

The quadratic model has gradient $g(x_k)+B_{k}p$ and positive-definite Hessian matrix $B_k$. Its unique minimizer is thereforeIf $g(x_k)\ne 0$, then $p_k\ne 0$ and $g(x_k{)}^T{p}_k=-p_k^T{B}_k{p}_k<0$, proving that it is a descent direction.

Backtracking line search starts with a trial step and repeatedly reduces it until the Armijo condition holds. Part (ii) guarantees finite termination because sufficiently small steps are acceptable. The test guarantees a specified positive fraction of the decrease predicted by the linear model, namely $f(x_k+\alpha p_k)\le f(x_k)+\alpha \beta g(x_k{)}^T{p}_k<f(x_k)$ for an accepted positive step. It does not require solving the line-minimization problem exactly.

If the first trial is accepted, ${\alpha }_k={\alpha }_{\mathrm{init}}$. Otherwise the preceding trial was rejected and so exceeded ${\alpha }_{\max }$, since part (ii) guarantees acceptance below that threshold. The accepted step, reduced by the factor $\tau$, consequently exceeds $\tau {\alpha }_{\max }$. In either case,The angle condition $-g_k^T{p}_k\ge \epsilon \Vert g_k{\Vert }_2\Vert p_k{\Vert }_2$ gives ${\alpha }_{\max }\ge 2\epsilon (1-\beta )\Vert g_k{\Vert }_2/(\gamma \Vert p_k{\Vert }_2)$. Hence the backtracking line search has the lower bound

From $B_k{p}_k=-g_k$ and the matrix 2-norm bound, $\Vert g_k{\Vert }_2\le {\kappa }_b\Vert p_k{\Vert }_2$, so $\Vert p_k{\Vert }_2\ge \Vert g_k{\Vert }_2/{\kappa }_b$. Let $C=2\tau \epsilon (1-\beta )/\gamma >0$. The summand in the supplied decrease bound satisfiesThus, with the fixed positive constant $c=\beta \epsilon \min ({\alpha }_{\mathrm{init}}/{\kappa }_b,C)$,The nonnegative series therefore converges, forcing its terms to zero:The argument uses the given angle condition and upper bound on $\Vert B_k{\Vert }_2$; it does not need an additional uniform lower eigenvalue bound on $B_k$.