Solution (source code)

= Solution

A <strict Lyapunov function> is a <continuously differentiable function> $V$ with $V(0)=0$, $V(x)>0$ off the origin, and $\nabla V\cdot f<0$ off the origin in the domain. Its <compact> <sublevel sets> inside the domain are forward invariant and force convergence to the equilibrium. The <domain of stability>, or <basin of attraction>, consists of initial points whose forward solutions exist for all positive time and converge to the <fixed point>.

For $V=(x^2+y^2)/2$, direct differentiation and factorization give
$$
\dot V=-2(x^2+y^2)+x^4+5x^2y^2+4y^4=(x^2+y^2)(-2+x^2+4y^2).
$$
If $R^2=x^2+y^2<1/2$, then $x^2+4y^2\le4R^2<2$, so $\dot V<0$ off zero. For any initial point in that disc, choose a slightly larger <closed disc> still inside it. The <vector field> points inward on its boundary; the trajectory is bounded, exists globally, and the strict Lyapunov decrease forces its limit to be zero. Thus the origin is <asymptotically stable> and the whole <open disc> lies in its basin.

If $R^2>2$, then $x^2+4y^2\ge R^2>2$, so $\dot V>0$. A solution starting there cannot cross inward through $R^2=2$ and cannot approach zero. Therefore
$$
\boxed{\{x^2+y^2<1/2\}\subseteq\mathcal B(0)\subseteq\{x^2+y^2\le2\}.}
$$