Solution (source code)

= Solution

For $H=x^2+y^2$, $\dot H=2\mu x^2(1-ax^2-y^2)$. It points outward on a sufficiently small circle $r<1/\sqrt a$ and inward on a circle $r>1$. The intervening annulus is compact, forward invariant, and contains no equilibrium. <Poincaré-Bendixson theorem> therefore gives a periodic orbit.