 Solution

The chain rule along the specified straight line givesHere equality of mixed partial derivatives follows from smoothness. Apply Taylor theorem in the single variable $t$:Writing $h_1=x_1-a=\alpha t$ and $h_2=x_2-b=\beta t$ produces the linear and quadratic termswith all displayed partial derivatives evaluated at $(a,b)$. For a remainder valid uniformly over directions, apply the Taylor formula with integral remainder to $s\mapsto f((a,b)+sh)$:Continuity of the Hessian matrix makes the difference between the integral and $\frac{1}{2}{h}^{T}H(a,b)h$ be $o(|h{|}^2)$. The printed ellipsis is therefore justified as a finite Taylor expansion with a remainder. Smoothness alone does not assert convergence of an infinite Taylor series.

At a stationary point, the linear term vanishes. The symmetric Hessian matrix satisfiesIf $H_{11}>0$ and $\det H>0$, this is a positive-definite quadratic form. Its minimum over the unit circle is a positive number $c$, so $h^{T}Hh\ge c|h{|}^2$. The Taylor remainder is smaller than $c|h{|}^2/4$ for sufficiently small $h$, proving $f((a,b)+h)>f(a,b)$ for nonzero nearby $h$. Thus the stationary point is a strict local minimum.

For the given polynomial, the gradient equations reduce toThe stationary points are $(0,0),(1,-1),(-1,1)$, andAt both points with $x_1=\pm 1$, we have $H_{11}=10>0$ and $\det H=16>0$. ConsequentlyAt the origin the determinant of the Hessian matrix is $-8$, giving a saddle point. An independent check is the identity $f=(x_1^2-1{)}^2+(x_1+x_2{)}^2-1$, which proves that the two displayed local minima are also the only global minima.