Two reasons to prefer more advanced methods for smooth, high-accuracy optimization are:
- Slow progress on ill-conditioned problems. On an elongated quadratic level set, successive negative gradients can zigzag across the narrow valley. Convergence can be only linear with a factor close to one, and line search cannot remove the underlying poor scaling.
- Failure to retain curvature information. Each new step again uses only the current gradient. Newton's method and quasi-Newton methods exploit curvature to obtain substantially faster local convergence under suitable hypotheses, often compensating for their more expensive steps.
The question's blanket wording is too strong: the steepest descent method and related gradient algorithms are used in practice, especially when problem size, cheap gradients or simplicity make curvature-based alternatives costly. The objections describe limitations, not universal nonuse.
Articles by others on the same topic
There are currently no matching articles.