Hessian matrix Created 2026-09-24 Updated 2026-10-14
The Hessian matrix describes how the slope of a function changes in different directions. For a real-valued function with continuous second partial derivatives on an open subset of , it isThe symmetry of second derivatives makes a symmetric matrix under these hypotheses. It is also the Jacobian matrix of the gradient: near , a small displacement changes by . Thus the gradient measures the current slope, while the Hessian measures its local variation. [ref]
For example, take the quadratic formIts gradient and Hessian matrix areThe only critical point is the origin. Completing the square gives , so the origin is a strict global minimum. The eigenvalues and are both positive, recovering the same conclusion from the positive definiteness of the Hessian. Its eigenvectors identify the rotated directions in which the quadratic form has its smallest and largest bending.
For a fixed vector , restriction to a straight line gives the directional interpretationThis is the quadratic form associated with the Hessian matrix. For a unit eigenvector , it equals the corresponding eigenvalue. Opposite signs therefore mean that some directions bend upwards and others downwards. The second-derivative test turns this information into a classification when the gradient vanishes.
The same object is useful for computation. Newton's method in optimization chooses a displacement by solving when the Hessian matrix is an invertible matrix; this makes the gradient of the local quadratic model vanish. For the displayed , the equation gives , so one exact step reaches the minimum from any starting point. For a general function, a Hessian with opposite-sign eigenvalues or a large condition number need not give a useful minimizing step, and the local minimum need not be a global minimum.
The name honors Otto Hesse. [ref]
https://web.mit.edu/course/other/wwmath/vectorc/minmax/hessian.html. MIT World Web Math: the Hessian and its namesake
An older geometric use is the Hessian criterion for flexes of a plane cubic, connecting a determinant of second partial derivatives with inflection points of an algebraic curve. Hesse's 1844 paper Über die Wendepuncte der Curven dritter Ordnung is an early source on that geometric problem. [ref]https://www.degruyterbrill.com/document/doi/10.1515/crll.1844.28.97/html. Hesse (1844), Journal für die reine und angewandte Mathematik 28, pp. 97–107: publisher's bibliographic record
Multivariable Newton iteration 2026-10-25
The finite-dimensional Newton root-finding iteration for a vector equation . At a point with invertible Jacobian matrix, solve the displayed linear system. For , its Jacobian matrix is the Hessian matrix, giving Newton's method in optimization. The update solves the affine first-order approximation ; it can converge to any nondegenerate critical point, including a saddle point.
Newton map in optimization 2026-10-24
For a twice continuously differentiable real function, the Newton map sends an input to the next iterate of Newton's method in optimization. It is defined where the Hessian matrix is invertible. Iterating this map seeks a zero of a function of the gradient; additional information is needed to determine whether a resulting stationary point is a local minimum.
Newton minimization of the square-root norm Created 2026-10-24 Updated 2026-10-31
For the Euclidean norm on , the regularized radial function has gradient and Hessian . The Cauchy-Schwarz inequality makes its Hessian a positive-definite matrix, so is strictly convex with unique minimum at zero. Its inverse matrix is : multiplication verifies this using and . Substitution in Newton's method in optimization therefore gives . Consequently : nonzero starts in the open unit ball have exact cubic convergence, starts on the unit sphere alternate between opposite points, and starts outside it diverge. This example separates strict convexity from global convergence of an undamped iteration.
Newton step 2026-10-24
A Newton step for Newton's method in optimization is the displacement solving the displayed linear system, where is the Hessian matrix and the gradient. The next iterate is . This makes the first order Taylor expansion of the gradient at the new point vanish. An invertible Hessian matrix determines a unique step; it need not give a descent direction when the Hessian matrix is indefinite.
For an unconstrained differentiable function of two real variables, an interior local minimum must solve the simultaneous stationary point equationsApply Newton's method in optimization to this gradient equation. The Jacobian matrix of the gradient is the Hessian matrixAt , solve the linear system , then set . When the Hessian matrix is invertible, this isThis follows by setting the first order Taylor expansion equal to zero. Solving the linear system avoids explicitly computing a matrix inverse. A stationary point need not be a local minimum: for example, a positive-definite matrix suffices to classify a stationary point as a strict local minimum, whereas an indefinite Hessian matrix identifies a saddle point of a scalar function.
To find a critical point, apply multivariable Newton iteration to the gradient, rather than to the scalar value . The gradient and Hessian matrix areWrite the correction as . At , Newton's method in optimization requiresSubtracting the second equation from the first gives , so and . HenceThe gradient there isA critical point has not been found after one iteration.
At the alternative starting point , the Hessian matrix and gradient areThe correction would need to satisfy and simultaneously. This linear system is inconsistent. The standard iteration cannot take even its first step: the Hessian matrix is a singular matrix, and the required correction does not exist. This is a Newton correction compatibility condition failure, not merely an ambiguity between several corrections.
The update in Newton's method in optimization is . Substituting the previous formulas givesThus the distance from the unique minimum obeys , where . In particular,For , the error ratio is exactly , proving exact cubic convergence. If , the initial point already attains the minimum. All updates are defined because the Hessian is invertible everywhere. This is Newton minimization of the square-root norm; its special update also shows why the given initial condition matters: produces an alternating two-cycle, and diverges.
Past exam of the year 1 of the computer science course of the University of Oxford 2021 Discrete mathematics, continuous mathematics, and probability 6 b i Solution Created 2026-10-25 Updated 2026-10-31
A practical choice is a BFGS method, using exact gradients and a line search. At dimension , its -by- inverse-Hessian matrix approximation is small, whereas forming a million rank-one contributions to the exact Hessian matrix is expensive. If , the explicit derivatives areThe positive-definite Hessian matrix and coercivity show that the objective has one global minimizer.
For numerical stability one can minimize instead, which has exactly the same minimizer. Put , , and . Then the logarithmic gradient of an exponential squared-distance sum givesEvaluate the weights as to avoid floating-point overflow. A quasi-Newton method step iswhere is the positive-definite inverse-Hessian matrix approximation updated by BFGS, and is chosen by a line search, for example one satisfying the Wolfe conditions. Under the standard smoothness and nondegeneracy assumptions at this strongly convex minimizer, full-memory BFGS has local superlinear convergence. An exact Newton's method in optimization would have local quadratic convergence, but pays for those much more expensive exact second derivatives.[ref]
Quasi-Newton Methods by NEOS Guide. Opening construction and BFGS update: approximate Hessians from gradients; dense update cost of order d squared
For Newton's method in optimization, let . Its second row equation is , so . Therefore . The given first coordinate yieldsThe Newton step is defined where ; both coordinates after any defined step are equal.
The full, undamped Newton's method in optimization step isThe eigenvector identity gives . Consequently for every starting point. This is a one-step Newton solution of a quadratic stationary equation: its gradient is an affine function, so a single exact Newton solve resolves it completely.
Gradient descent uses only the current gradient and a suitable step size; its additional storage is order . It is attractive for very large or when low per-iteration cost matters most. For a smooth strongly convex function, standard step choices give linear convergence, with a rate sensitive to the condition number. Mere convexity gives a global-minimum interpretation but does not alone imply a linear rate; for nonconvex objectives the usual guarantee concerns stationary points.
BFGS uses successive gradients to update an approximation to inverse curvature, incorporating information that can correct poor scaling. Full BFGS uses order storage and update work. With suitable line searches and local smoothness and positive-curvature assumptions near a nondegenerate minimum, it has superlinear convergence. This often favours it in moderate dimensions when gradients are not extremely cheap relative to the extra work.
Neither gradient descent nor BFGS requires an explicitly evaluated Hessian matrix. Thus an expensive Hessian matrix is compatible with both, rather than a reason to reject BFGS. If an accurate Hessian matrix and its factorization are exceptionally cheap, Newton's method in optimization becomes an additional reasonable option. Convexity helps identify a stationary point as a global minimum; without it, both methods need suitable descent and curvature safeguards. The BFGS update learns curvature from the Quasi-Newton secant equation.[ref]
Quasi-Newton Methods by NEOS Guide. Quasi-Newton condition and the BFGS member of the low-rank updating family.
Quasi-Newton method Created 2026-10-24 Updated 2026-10-31
A Quasi-Newton method replaces the Hessian matrix in Newton's method in optimization by a matrix learned from successive gradients. Solving gives a descent direction whenever is a positive-definite symmetric matrix and the gradient is nonzero. After choosing a step size, one updates to satisfy the Quasi-Newton secant equation, rather than calculating every second derivative again.
For a twice continuously differentiable real function, define the Newton map in optimization wherever its Hessian matrix is invertible. Every stationary point in this domain is a fixed point, because its gradient is zero. This includes a local maximum and a saddle point of a scalar function, as well as a local minimum. Thus Newton's method in optimization needs a separate minimum classification or a method for ensuring descent when it is used for minimization problem.
For example, has a stationary point at , with Hessian matrixIts eigenvalues are and , so it is a saddle point of a scalar function, but starting Newton's method in optimization there gives a constant sequence. Its value is , whereas the global minimum zero is attained at and .
