Hessian matrix Created 2026-09-24 Updated 2026-10-14
The Hessian matrix describes how the slope of a function changes in different directions. For a real-valued function with continuous second partial derivatives on an open subset of , it is
The symmetry of second derivatives makes a symmetric matrix under these hypotheses. It is also the Jacobian matrix of the gradient: near , a small displacement changes by . Thus the gradient measures the current slope, while the Hessian measures its local variation. [ref]
For example, take the quadratic form
Its gradient and Hessian matrix are
The only critical point is the origin. Completing the square gives , so the origin is a strict global minimum. The eigenvalues and are both positive, recovering the same conclusion from the positive definiteness of the Hessian. Its eigenvectors identify the rotated directions in which the quadratic form has its smallest and largest bending.
Figure 1.
Positive, negative and mixed Hessian eigenvalues produce different critical-point geometry
. Contour plots of , , and . All three have a critical point at the origin; the first two have elliptical contours, while the last has positive and negative sectors meeting along its zero contours. Details.
For a fixed vector , restriction to a straight line gives the directional interpretation
This is the quadratic form associated with the Hessian matrix. For a unit eigenvector , it equals the corresponding eigenvalue. Opposite signs therefore mean that some directions bend upwards and others downwards. The second-derivative test turns this information into a classification when the gradient vanishes.
The same object is useful for computation. Newton's method in optimization chooses a displacement by solving when the Hessian matrix is an invertible matrix; this makes the gradient of the local quadratic model vanish. For the displayed , the equation gives , so one exact step reaches the minimum from any starting point. For a general function, a Hessian with opposite-sign eigenvalues or a large condition number need not give a useful minimizing step, and the local minimum need not be a global minimum.
The name honors Otto Hesse. [ref] An older geometric use is the Hessian criterion for flexes of a plane cubic, connecting a determinant of second partial derivatives with inflection points of an algebraic curve. Hesse's 1844 paper Über die Wendepuncte der Curven dritter Ordnung is an early source on that geometric problem. [ref]
The finite-dimensional Newton root-finding iteration for a vector equation . At a point with invertible Jacobian matrix, solve the displayed linear system. For , its Jacobian matrix is the Hessian matrix, giving Newton's method in optimization. The update solves the affine first-order approximation ; it can converge to any nondegenerate critical point, including a saddle point.
For a twice continuously differentiable real function, the Newton map sends an input to the next iterate of Newton's method in optimization. It is defined where the Hessian matrix is invertible. Iterating this map seeks a zero of a function of the gradient; additional information is needed to determine whether a resulting stationary point is a local minimum.
Newton minimization of the square-root norm Created 2026-10-24 Updated 2026-10-31
For the Euclidean norm on , the regularized radial function has gradient and Hessian . The Cauchy-Schwarz inequality makes its Hessian a positive-definite matrix, so is strictly convex with unique minimum at zero. Its inverse matrix is : multiplication verifies this using and . Substitution in Newton's method in optimization therefore gives . Consequently : nonzero starts in the open unit ball have exact cubic convergence, starts on the unit sphere alternate between opposite points, and starts outside it diverge. This example separates strict convexity from global convergence of an undamped iteration.
Newton step 2026-10-24
A Newton step for Newton's method in optimization is the displacement solving the displayed linear system, where is the Hessian matrix and the gradient. The next iterate is . This makes the first order Taylor expansion of the gradient at the new point vanish. An invertible Hessian matrix determines a unique step; it need not give a descent direction when the Hessian matrix is indefinite.
For an unconstrained differentiable function of two real variables, an interior local minimum must solve the simultaneous stationary point equations
Apply Newton's method in optimization to this gradient equation. The Jacobian matrix of the gradient is the Hessian matrix
At , solve the linear system , then set . When the Hessian matrix is invertible, this is
This follows by setting the first order Taylor expansion equal to zero. Solving the linear system avoids explicitly computing a matrix inverse. A stationary point need not be a local minimum: for example, a positive-definite matrix suffices to classify a stationary point as a strict local minimum, whereas an indefinite Hessian matrix identifies a saddle point of a scalar function.
To find a critical point, apply multivariable Newton iteration to the gradient, rather than to the scalar value . The gradient and Hessian matrix are
Write the correction as . At , Newton's method in optimization requires
Subtracting the second equation from the first gives , so and . Hence
The gradient there is
A critical point has not been found after one iteration.
At the alternative starting point , the Hessian matrix and gradient are
The correction would need to satisfy and simultaneously. This linear system is inconsistent. The standard iteration cannot take even its first step: the Hessian matrix is a singular matrix, and the required correction does not exist. This is a Newton correction compatibility condition failure, not merely an ambiguity between several corrections.
The update in Newton's method in optimization is . Substituting the previous formulas gives
Thus the distance from the unique minimum obeys , where . In particular,
For , the error ratio is exactly , proving exact cubic convergence. If , the initial point already attains the minimum. All updates are defined because the Hessian is invertible everywhere. This is Newton minimization of the square-root norm; its special update also shows why the given initial condition matters: produces an alternating two-cycle, and diverges.
A practical choice is a BFGS method, using exact gradients and a line search. At dimension , its -by- inverse-Hessian matrix approximation is small, whereas forming a million rank-one contributions to the exact Hessian matrix is expensive. If , the explicit derivatives are
The positive-definite Hessian matrix and coercivity show that the objective has one global minimizer.
For numerical stability one can minimize instead, which has exactly the same minimizer. Put , , and . Then the logarithmic gradient of an exponential squared-distance sum gives
Evaluate the weights as to avoid floating-point overflow. A quasi-Newton method step is
where is the positive-definite inverse-Hessian matrix approximation updated by BFGS, and is chosen by a line search, for example one satisfying the Wolfe conditions. Under the standard smoothness and nondegeneracy assumptions at this strongly convex minimizer, full-memory BFGS has local superlinear convergence. An exact Newton's method in optimization would have local quadratic convergence, but pays for those much more expensive exact second derivatives.[ref]
For Newton's method in optimization, let . Its second row equation is , so . Therefore . The given first coordinate yields
The Newton step is defined where ; both coordinates after any defined step are equal.
The full, undamped Newton's method in optimization step is
The eigenvector identity gives . Consequently for every starting point. This is a one-step Newton solution of a quadratic stationary equation: its gradient is an affine function, so a single exact Newton solve resolves it completely.
Gradient descent uses only the current gradient and a suitable step size; its additional storage is order . It is attractive for very large or when low per-iteration cost matters most. For a smooth strongly convex function, standard step choices give linear convergence, with a rate sensitive to the condition number. Mere convexity gives a global-minimum interpretation but does not alone imply a linear rate; for nonconvex objectives the usual guarantee concerns stationary points.
BFGS uses successive gradients to update an approximation to inverse curvature, incorporating information that can correct poor scaling. Full BFGS uses order storage and update work. With suitable line searches and local smoothness and positive-curvature assumptions near a nondegenerate minimum, it has superlinear convergence. This often favours it in moderate dimensions when gradients are not extremely cheap relative to the extra work.
Neither gradient descent nor BFGS requires an explicitly evaluated Hessian matrix. Thus an expensive Hessian matrix is compatible with both, rather than a reason to reject BFGS. If an accurate Hessian matrix and its factorization are exceptionally cheap, Newton's method in optimization becomes an additional reasonable option. Convexity helps identify a stationary point as a global minimum; without it, both methods need suitable descent and curvature safeguards. The BFGS update learns curvature from the Quasi-Newton secant equation.[ref]
Quasi-Newton method Created 2026-10-24 Updated 2026-10-31
A Quasi-Newton method replaces the Hessian matrix in Newton's method in optimization by a matrix learned from successive gradients. Solving gives a descent direction whenever is a positive-definite symmetric matrix and the gradient is nonzero. After choosing a step size, one updates to satisfy the Quasi-Newton secant equation, rather than calculating every second derivative again.
For a twice continuously differentiable real function, define the Newton map in optimization wherever its Hessian matrix is invertible. Every stationary point in this domain is a fixed point, because its gradient is zero. This includes a local maximum and a saddle point of a scalar function, as well as a local minimum. Thus Newton's method in optimization needs a separate minimum classification or a method for ensuring descent when it is used for minimization problem.
For example, has a stationary point at , with Hessian matrix
Its eigenvalues are and , so it is a saddle point of a scalar function, but starting Newton's method in optimization there gives a constant sequence. Its value is , whereas the global minimum zero is attained at and .