 Solution

Put $y=\dot{x}$. The first-order system isAt an equilibrium point of a dynamical system, $y=0$ and $x(1-x^2{)}^2=0$, so the three fixed points are

DefineThen $U^{\prime }(x)=2x(1-x^2{)}^2=-F(x)$ andMoreover,so $E$ is positive definite at the origin. It is therefore a damped mechanical energy as a Lyapunov function, and the First Lyapunov theorem proves that the origin is Lyapunov stable.

Since $U(\pm 1)=1/3$, choose a compact energy sublevel $\{E\le c\}$ with $0<c<1/3$. In this set, $\dot{E}=0$ means $y=0$. A trajectory can remain in $y=0$ only when $F(x)=0$, and the chosen sublevel excludes $x=\pm 1$. Thus the largest invariant subset of $\{\dot{E}=0\}$ is the origin. The LaSalle invariance principle proves that the origin is asymptotically stable.

Finally, $E$ is radially unbounded because its leading terms are $x^6/3+y^2/2$. Every forward trajectory is therefore bounded. LaSalle's principle puts its omega-limit set insideAn omega-limit set of a bounded continuous trajectory is nonempty and connected. A connected subset of this three-point set is a singleton, so every trajectory has precisely one of the three fixed points as its omega-limit set.