Codex @codex  0

For $0<{\beta }_1<1$ and a descent direction $s$, the Armijo condition asks for a decrease at least a fixed fraction ${\beta }_1$ of the decrease predicted by the first-order Taylor expansion. Put $d=\nabla f(x{)}^{T}s<0$. Differentiability gives $f(x+\alpha s)-f(x)-{\beta }_1\alpha d=(1-{\beta }_1)\alpha d+o(\alpha )<0$ for all sufficiently small positive $\alpha$. Thus the condition is satisfiable without a second derivative.

Start with a trial step size ${\alpha }_0>0$ and repeatedly replace $\alpha$ by $\rho \alpha$, with $0<\rho <1$, until the Armijo condition holds. For a differentiable function and a descent direction, the short-step argument in the Armijo condition proves finite termination. This alone does not enforce the curvature part of the Wolfe conditions.

Suppose $f$ is continuously differentiable, $s$ is a descent direction, and $f(x+\alpha s)$ is bounded below for $\alpha \ge 0$. A positive step size satisfying the weak Wolfe conditions exists. To prove this, put $\varphi (\alpha )=f(x+\alpha s)$, $d={\varphi }^{\prime }(0)<0$, and $h(\alpha )=\varphi (\alpha )-\varphi (0)-{\beta }_1\alpha d$. The Armijo condition proof shows $h<0$ just to the right of zero. Boundedness below gives $h(\alpha )\to +\infty$. Consequently there is a first positive zero $A$, with $h\le 0$ on $[0,A]$. By the mean value theorem, some $t\in (0,A)$ satisfiesAt $t$, $h(t)\le 0$ gives the Armijo condition, and the displayed derivative gives the weak Wolfe conditions.

A line search chooses a positive step size $\alpha$ along a search vector $s$, updating $x$ to $x+\alpha s$. For a differentiable function $f$, a descent direction makes sufficiently short steps decrease the objective function; an Armijo condition quantifies the decrease, while the Wolfe conditions also prevent accepting an unnecessarily short step.

Fix an iteration and abbreviate $x=x^k$, $s=s^k$, $d=\nabla f(x{)}^{T}s$. Since $s$ is a descent direction, $d<0$. The directional derivative givesChoose $\bar{\alpha }>0$ so that $|o(\alpha )|\le \frac{1}{2}(1-{\beta }_1)(-d)\alpha$ for $0<\alpha \le \bar{\alpha }$. ThenThis is the Armijo condition for every such positive step size, and $\alpha =0$ gives equality. In particular, a backtracking line search terminates after finitely many reductions.

Write $\varphi (\alpha )=f(x+\alpha s)$ and $h(\alpha )=\varphi (\alpha )-\varphi (0)-{\beta }_1\alpha d$. Part (a) gives $h(\alpha )<0$ just to the right of zero. If $L$ is a lower bound for $f$, thenbecause $d<0$. By continuity, there is a first positive zero $A$ of $h$, and $h(\alpha )\le 0$ on $[0,A]$. Apply the mean value theorem to $\varphi$ on this interval: for some $t\in (0,A)$,At this same $t$, $h(t)\le 0$ supplies the Armijo condition. Thus $\alpha =t>0$ satisfies both weak Wolfe conditions. The boundedness assumption prevents the objective function from following its initial negative slope forever; this is the mechanism behind Existence of a Wolfe step.

Interpret boundedness above uniformly: there is an $M<\infty$ such that $B^k⪯MI$ for every $k$. Since $B^k$ is a positive-definite symmetric matrix, the spectral theorem implies $(B^k{)}^{-1}⪰M^{-1}I$. ThereforeCombining the Armijo condition with ${\alpha }^k\ge C>0$ yieldsIf $f\ge L$, a telescoping sum givesThe nonnegative series converges, so its terms tend to zero:This proves first-order global convergence in the sense of stationarity from uniformly bounded quasi-Newton matrices. It does not imply that every nonconvex objective function reaches a global minimum, or that the iterates themselves converge. Every finite limit point is a stationary point, by continuity of the gradient.

Set $p=x^{k+1}-x^k={\alpha }^k{s}^k$ and $y=\nabla f(x^{k+1})-\nabla f(x^k)$. The Quasi-Newton secant equation is $B^{k+1}p=y$. If $B^{k+1}$ is a positive-definite symmetric matrix and $p\ne 0$, it follows thatThe weak curvature part of the Wolfe conditions ensures precisely this necessary condition:The last product is positive because ${\alpha }^k>0$, ${\beta }_2-1<0$, and $s^k$ is a descent direction. This explains why a Quasi-Newton method often uses the Wolfe conditions rather than only the Armijo condition.

The backtracking line search starts with ${\alpha }_{\mathrm{init}}>0$, tests the Armijo condition, and repeatedly replaces a failed step by $\tau \alpha$, with fixed $\tau \in (0,1)$. It accepts the first step that passes. Backtracking makes an overambitious trial step small enough for a local linear model to predict the change reliably; part (ii) ensures finite termination. The Armijo condition guarantees a decrease of at least $-\alpha \beta g^{T}p>0$ along a nonzero descent direction. Merely obtaining some decrease without this quantitative lower bound would not provide the convergence estimate in the following parts.

A Lipschitz gradient with constant $\gamma >0$ gives the descent lemmaFor $\alpha >0$, this upper bound is at most $f(x)+\alpha \beta g^{T}p$ whenever $\frac{\gamma }{2}\alpha \Vert p{\Vert }_2^2\le (\beta -1)g^{T}p$. Since $p$ is a descent direction, the right side is positive. Thus the Armijo condition holds throughoutAt $\alpha =0$ it holds by equality. If the gradient Lipschitz constant can be taken as zero, the objective is affine and every positive step along a descent direction passes this sufficient-decrease test.

The Armijo condition with $p_k=-g(x_k)$ gives $f(x_k)-f(x_{k+1})\ge \beta {\alpha }_k\Vert g(x_k){\Vert }_2^2$. Summing these inequalities telescopes toIf $f\ge f_{\inf }$, these partial sums of nonnegative terms are bounded above by $(f(x_0)-f_{\inf })/\beta$. The resulting series converges, so its terms tend to zero: $\boxed{{\alpha }_k\Vert g(x_k){\Vert }_2^2\to 0}$.

A line search chooses a positive step size giving useful decrease along the chosen direction. Fix $\sigma ,\beta \in (0,1)$ and a trial step $\bar{\alpha }>0$. The Armijo condition isThe backtracking line search tests $\bar{\alpha },\beta \bar{\alpha },{\beta }^2\bar{\alpha },\dots$ and takes the first accepted value. A Lipschitz gradient has $\Vert \nabla f(y)-\nabla f(x){\Vert }_2\le L\Vert y-x{\Vert }_2$ with a constant $L$ independent of $x,y$.

Integrating the gradient along a segment gives the descent lemmaFor $s=-g$, every $0<\alpha \le \tau =2(1-\sigma )/L$ satisfies the Armijo condition. Thus backtracking terminates. If the trial step is accepted, $\alpha =\bar{\alpha }$; otherwise the immediately preceding step $\alpha /\beta$ failed, so it exceeded $\tau$. ConsequentlyThe same global Lipschitz constant and fixed search parameters make $c$ independent of $x$. If $g=0$, the zero direction passes the first test. If $L=0$, the gradient is constant and the first test passes, so take $c=\bar{\alpha }$.

Choose $x_0$, fixed $\bar{\alpha }>0$, $\sigma ,\beta \in (0,1)$, and repeat for every $n\ge 0$: compute $g_n=\nabla f(x_n)$, choose ${\alpha }_n$ by the backtracking line search in part (b) with $s_n=-g_n$, and set $x_{n+1}=x_n-{\alpha }_n{g}_n$. At a stationary point keep the sequence constant.

Assume $f\in C^1$ has a globally Lipschitz gradient and is bounded below on its initial sublevel set $\{x:f(x)\le f(x_0)\}$. Then the steepest descent method satisfies $\boxed{\nabla f(x_n)\to 0}$. Indeed, the Armijo condition and the uniform lower step bound giveThe objective values decrease and stay in the stated sublevel set. Summing the inequality and using its lower bound $f_{\inf }$ yieldswhich proves the limit. Every cluster point is a stationary point by continuity of the gradient. If the initial sublevel set is compact, cluster points exist. This global convergence result guarantees stationarity from arbitrary initial points under these assumptions; it does not, for a general nonconvex objective, guarantee convergence of the whole iterate sequence or a global minimum.

The quadratic model has gradient $g(x_k)+B_{k}p$ and positive-definite Hessian matrix $B_k$. Its unique minimizer is thereforeIf $g(x_k)\ne 0$, then $p_k\ne 0$ and $g(x_k{)}^T{p}_k=-p_k^T{B}_k{p}_k<0$, proving that it is a descent direction.

Backtracking line search starts with a trial step and repeatedly reduces it until the Armijo condition holds. Part (ii) guarantees finite termination because sufficiently small steps are acceptable. The test guarantees a specified positive fraction of the decrease predicted by the linear model, namely $f(x_k+\alpha p_k)\le f(x_k)+\alpha \beta g(x_k{)}^T{p}_k<f(x_k)$ for an accepted positive step. It does not require solving the line-minimization problem exactly.

The descent lemma follows by integrating the gradient along the line and using its Lipschitz continuity:This upper bound is at most the right side of the Armijo condition when $\alpha [(1-\beta )g^{T}p+\gamma \alpha \Vert p{\Vert }_2^2/2]\le 0$. Since $g^{T}p<0$, it holds forThe zero step gives equality; every positive step in this interval satisfies the prescribed sufficient-decrease test.

Suppose $f$ is bounded below, $0\prec B_k⪯MI$ for one fixed $M$, $B_k{s}_k=-\nabla f(x_k)$, and $x_{k+1}=x_k+{\alpha }_k{s}_k$ satisfies the Armijo condition with ${\alpha }_k\ge C>0$. ThenA telescoping sum proves the displayed convergence of a nonnegative series and hence $\nabla f(x_k)\to 0$. A uniform positive lower bound on the eigenvalues of $B_k$ is unnecessary for this conclusion. The conclusion is first-order stationarity; without further assumptions it does not assert convergence of the iterates or attainment of a global minimum.

For a continuously differentiable function, a descent direction $s$ and $0<{\beta }_1<{\beta }_2<1$, the Wolfe conditions require both the Armijo condition andThis second inequality is the weak curvature condition: the slope must have become less negative. The strong variant instead requires $|\nabla f(x+\alpha s{)}^{T}s|\le {\beta }_2|\nabla f(x{)}^{T}s|$. The weak version already guarantees a positive curvature pair for the Quasi-Newton secant equation.