 Solution

For $H=x^2+y^2$, $\dot{H}=2\mu x^2(1-a{x}^2-y^2)$. It points outward on a sufficiently small circle $r<1/\sqrt{a}$ and inward on a circle $r>1$. The intervening annulus is compact, forward invariant, and contains no equilibrium. Poincare-Bendixson theorem therefore gives a periodic orbit.