= Solution
{c}
For a twice <continuously differentiable function>, an interior <local extremum> requires $f_x=f_y=0$. At such a <critical point>, put $D=f_{xx}f_{yy}-f_{xy}^2$. The <second-derivative test> gives a strict <local minimum> if $D>0$ and $f_{xx}>0$, a strict <local maximum> if $D>0$ and $f_{xx}<0$, and a <saddle point> if $D<0$. When $D=0$, the test is inconclusive. More generally the <Hessian matrix> is necessarily <positive semidefinite> at a <local minimum> and <negative semidefinite> at a <local maximum>, but these non-strict signs alone are not sufficient.
For this <polynomial>,
$$
f_x=2x+2y-y^2,\quad f_y=2x+2y-2xy,\quad
H=\begin{pmatrix}2&2-2y\\2-2y&2-2x\end{pmatrix}.
$$
Subtracting the two stationarity equations gives $y(2x-y)=0$. If $y=0$, then $x=0$; if $y=2x$, the remaining equation gives $x=0$ or $x=3/2$. Thus the only <critical points> are $(0,0)$ and $(3/2,3)$.
At $(3/2,3)$, $\det H=-18<0$, so it is a <saddle point>. At $(0,0)$ the determinant is zero, but along $y=-x$ the value is $-x^3$, which takes both signs arbitrarily near zero. Therefore
$$
\boxed{(0,0)\text{ and }(3/2,3)\text{ are both saddle points}.}
$$
In particular a vanishing <Hessian determinant> does not exclude a <saddle point>.
Back to article page
