Hessian matrix ($\mathrm{Hess}f$)

The Hessian matrix describes how the slope of a function changes in different directions. For a real-valued function $f$ with continuous second partial derivatives on an open subset of ${\mathbb{R}}^n$, it isThe symmetry of second derivatives makes $H_f$ a symmetric matrix under these hypotheses. It is also the Jacobian matrix of the gradient: near $x$, a small displacement $h$ changes $\nabla f$ by $H_f(x)h+o(\Vert h\Vert )$. Thus the gradient measures the current slope, while the Hessian measures its local variation. ^[ref]

For example, take the quadratic formIts gradient and Hessian matrix areThe only critical point is the origin. Completing the square gives $q=(x+y{)}^2+2y^2$, so the origin is a strict global minimum. The eigenvalues $4-2\sqrt{2}$ and $4+2\sqrt{2}$ are both positive, recovering the same conclusion from the positive definiteness of the Hessian. Its eigenvectors identify the rotated directions in which the quadratic form has its smallest and largest bending.

For a fixed vector $v$, restriction to a straight line gives the directional interpretationThis is the quadratic form associated with the Hessian matrix. For a unit eigenvector $v$, it equals the corresponding eigenvalue. Opposite signs therefore mean that some directions bend upwards and others downwards. The second-derivative test turns this information into a classification when the gradient vanishes.

The same object is useful for computation. Newton's method in optimization chooses a displacement $h$ by solving $H_f(x)h=-\nabla f(x)$ when the Hessian matrix is an invertible matrix; this makes the gradient of the local quadratic model vanish. For the displayed $q$, the equation gives $h=-(x,y{)}^T$, so one exact step reaches the minimum from any starting point. For a general function, a Hessian with opposite-sign eigenvalues or a large condition number need not give a useful minimizing step, and the local minimum need not be a global minimum.

The name honors Otto Hesse. ^[ref] An older geometric use is the Hessian criterion for flexes of a plane cubic, connecting a determinant of second partial derivatives with inflection points of an algebraic curve. Hesse's 1844 paper Über die Wendepuncte der Curven dritter Ordnung is an early source on that geometric problem. ^[ref]

Let $\varphi$ be a twice continuously differentiable local diffeomorphism, and suppose $\nabla f(\varphi (u_{\ast }))=0$. The chain rule gives the displayed transformation of the Hessian matrix: terms containing second partial derivatives of $\varphi$ are multiplied by first partial derivatives of $f$ and vanish at the critical point. By Sylvester's law of inertia, the numbers of positive, negative and zero eigenvalues are unchanged. Independently of nondegeneracy, the local diffeomorphism preserves the critical point's status as a local minimum, local maximum or saddle point of a scalar function, because it maps neighbourhoods to neighbourhoods.

Applying a Hessian matrix to a vector without explicitly storing all its entries. For $H=A^{\mathsf{T}}\Delta A$ with diagonal $\Delta$, evaluate $Hv=A^{\mathsf{T}}(\Delta (Av))$. This uses two matrix-vector multiplications and a diagonal scaling: $O(nd)$ arithmetic for dense $A\in {\mathbb{R}}^{n\times d}$, rather than forming the $d\times d$ Hessian matrix. Iterative second-order methods can use this operation, with safeguards for indefinite curvature when the objective is nonconvex.

For twice differentiable $f:\mathbb{R}\to \mathbb{R}$, constant vectors $a_i$ and constant scalars $y_i$, let $l(x)={\sum }_i{y}_{i}f(a_i^{\mathsf{T}}x)$, and let row $i$ of $A$ be $a_i^{\mathsf{T}}$. The chain rule givesThe second formula is the displayed matrix product because entry $(j,k)$ is the same weighted sum ${\sum }_i{y}_i{f}^{\prime \prime }(a_i^{\mathsf{T}}x)a_{ij}{a}_{ik}$. Signed diagonal factors do not ensure a positive-definite Hessian matrix. The factorization also gives an efficient matrix-free Hessian-vector product.

The determinant of the Hessian matrix. For a twice continuously differentiable function of two variables, it is $f_{xx}{f}_{yy}-f_{xy}^2$. A negative value at a critical point implies a saddle point; a positive value combines with the sign of $f_{xx}$ in the second-derivative test. A zero value is inconclusive.

For $F\in C^2({\mathbb{R}}^n)$ with bounded Hessian matrix, Taylor's formula along the line from zero to $p$ gives $|F(p)|\le |F(0)|+|DF(0)||p|+\frac{1}{2}\Vert D^{2}F{\Vert }_{\infty }|p{|}^2$. Its gradient obeys $|DF(p)|\le |DF(0)|+\Vert D^{2}F{\Vert }_{\infty }|p|$. Consequently ${\int }_{\Omega }F(Du)$ is finite for $u\in H^1(\Omega )$ when $\Omega$ has finite measure, and $DF(Du)\in L^2$.

For $f(x,y)=g(x)+h(y)$, a degenerate Hessian matrix can be supplemented by the first nonzero term of each one-variable displacement. Two negative leading even powers give a strict local maximum, two positive ones a strict local minimum, and opposite signs give a saddle. Looking along the coordinate directions and bounding the separated terms avoids an inconclusive quadratic test.

For an explicit contrast, considerBoth have zero gradient and the same zero Hessian matrix at the origin:Nevertheless, $f_{+}$ is positive at every nonzero point, so the origin is a strict global minimum. For $f_{-}$, the restrictions $f_{-}(t,0)=t^4>0$ and $f_{-}(0,t)=-t^4<0$ for $t\ne 0$ show that the origin is a saddle point of a scalar function. Identical quadratic information can conceal different geometry when higher-order terms are decisive.

A symmetric matrix $A$ is negative semidefinite when $v^{T}Av\le 0$ for every vector $v$. Equivalently, all its eigenvalues are nonpositive.

Let $f$ have continuous second partial derivatives near an interior critical point $x_{\ast }$ of an open subset of ${\mathbb{R}}^n$, with $\nabla f(x_{\ast })=0$. The Hessian matrix gives the following sufficient conditions:

A positive semidefinite or negative semidefinite Hessian matrix with a zero eigenvalue does not settle the classification. The higher-order test for separated extrema gives two quartic examples with exactly the same zero Hessian. These conditions apply to an interior critical point; a boundary optimum or a constrained optimization problem requires attention to the allowed directions. ^[ref]

The proof outline is the Taylor expansionFor a positive-definite matrix, the smallest eigenvalue supplies a positive bound proportional to $\Vert h{\Vert }^2$, which dominates the remainder for small enough $h$. Negate the function for the local maximum case. For opposite-sign eigenvalues, choose the two corresponding eigenvectors: the leading terms have opposite signs along their lines.

There are also necessary conditions. At an interior local minimum, the Hessian matrix is positive semidefinite; at an interior local maximum, it is negative semidefinite. Restriction to each straight line reduces this to the one-variable second derivative condition. Since the Laplacian is the trace of the Hessian, it is nonnegative at such a local minimum and nonpositive at such a local maximum. The sign of the Laplacian alone is insufficient: $x^2-2y^2$ has negative Laplacian but a saddle point of a scalar function at the origin.

Define $f(0,0)=0$ and, elsewhere,All first and second partial derivatives exist at the origin: $f_x=f_y=f_{xy}=f_{yx}=0$ and $f_{xx}=f_{yy}=2$. Thus its Hessian matrix there is the positive-definite matrix $2I$. Nevertheless $f(t,0)=t^2>0$ and $f(t,t)=-t^2<0$ for $t\ne 0$, so the origin is a saddle point of a scalar function. A valid second-derivative test needs a second-order Taylor expansion with a remainder $o(x^2+y^2)$; continuous second partial derivatives in a neighborhood are a sufficient hypothesis. Existence of those derivatives at the point alone does not supply this hypothesis.

The origin is a saddle point of a scalar function for every real parameter. For $\alpha >0$ four further critical points have $x^2=y^2=1/(2\alpha )$. Equal-sign coordinates give global maxima and opposite-sign coordinates give global minima, with values $\pm 1/(2e\alpha )$. The Hessian matrix there is $-4\alpha xy{e}^{-1}I$. Their distance from the origin is ${\alpha }^{-1/2}$, so as $\alpha \downarrow 0$ they escape every compact region. At nonpositive $\alpha$ the origin is the only critical point. This illustrates why the number of critical points on an unbounded domain can change without a finite degenerate critical point appearing.

A critical point of a real function with continuous second partial derivatives is nondegenerate when its Hessian matrix is an invertible matrix there, equivalently when none of its eigenvalues is zero. A positive-definite matrix gives a strict local minimum, a negative definite Hessian gives a strict local maximum, and opposite-sign eigenvalues give a saddle point of a scalar function. A nondegenerate critical point need not be an extremum: $x^2-y^2$ has Hessian matrix $\mathrm{diag}(2,-2)$, which is an invertible matrix.

Such critical points are the local building blocks of Morse theory.