= Solution
<Gradient descent> uses only the current <gradient> and a suitable <step size>; its additional storage is order $d$. It is attractive for very large $d$ or when low per-iteration cost matters most. For a smooth <strongly convex function>, standard step choices give <linear convergence>, with a rate sensitive to the <condition number>. Mere <convexity> gives a global-minimum interpretation but does not alone imply a linear rate; for nonconvex objectives the usual guarantee concerns stationary points.
<BFGS> uses successive <gradients> to update an approximation to inverse curvature, incorporating information that can correct poor scaling. Full <BFGS> uses order $d^2$ storage and update work. With suitable <line searches> and local smoothness and positive-curvature assumptions near a nondegenerate <minimum>, it has <superlinear convergence>. This often favours it in moderate dimensions when gradients are not extremely cheap relative to the extra work.
Neither <gradient descent> nor <BFGS> requires an explicitly evaluated <Hessian matrix>. Thus an expensive <Hessian matrix> is compatible with both, rather than a reason to reject <BFGS>. If an accurate <Hessian matrix> and its factorization are exceptionally cheap, <Newton's method in optimization> becomes an additional reasonable option. <Convexity> helps identify a stationary point as a global <minimum>; without it, both methods need suitable descent and curvature safeguards. The <BFGS> update learns curvature from the <Quasi-Newton secant equation>.<Quasi-Newton Methods by NEOS Guide>[Quasi-Newton condition and the BFGS member of the low-rank updating family.]{ref}
Back to article page
