 Solution

At a stationary point of a twice continuously differentiable function, write the Hessian matrix as $H=\left(\begin{array}{cc}A& B\\ B& C\end{array}\right)$ and its determinant as $D=AC-B^2$. The quadratic term in the Taylor expansion determines the local nature whenever $H$ is nonsingular.

If $D>0$, its two eigenvalues have the same sign. With $A>0$ the quadratic form is positive definite, giving a strict local minimum; with $A<0$ it is negative definite, giving a strict local maximum. If $D<0$, the eigenvalues have opposite signs and the point is a saddle, with nearby increasing and decreasing directions. ThusIf $D=0$, the test is inconclusive. For example, $x^4+y^4$, $-x^4-y^4$ and $x^4-y^4$ all have zero gradient and zero Hessian matrix at the origin, yet give a minimum, a maximum and a saddle respectively. One must inspect higher-order terms or use another argument in such a case.