 Solution

Use the nearly conserved oscillator energy $E=({\dot{x}}^2+x^2)/2$. The differential equation givesFor a leading-order orbit $x=A\cos \theta$, $\dot{x}=-A\sin \theta$, the cycle averages are $⟨{\dot{x}}^4⟩=3A^4/8$, $⟨x^3\dot{x}⟩=0$, $⟨x^4{\dot{x}}^2⟩=A^6/16$, and $⟨{\dot{x}}^2⟩=A^2/2$. The energy balance method therefore givesThe two nonzero simple roots give the leading limit cycles $x\simeq \cos (t+{\theta }_0)$ and $x\simeq \sqrt{2}\cos (t+{\theta }_0)$. Below amplitude $1$ the energy increases, between $1$ and $\sqrt{2}$ it decreases, and above $\sqrt{2}$ it increases. Hence the amplitude-$1$ cycle is stable and the amplitude-$\sqrt{2}$ cycle unstable for positive small $\epsilon$.

The linearized averaged radial exponents are $-\epsilon /2$ and $\epsilon$, respectively, consistent with nontrivial Floquet multipliers $e^{-\pi \epsilon +O({\epsilon }^2)}$ and $e^{2\pi \epsilon +O({\epsilon }^2)}$. The nonzero derivatives at these averaged zeros also justify persistence of nearby cycles for sufficiently small $\epsilon$.