Solution (source code)

= Solution

Use the nearly conserved oscillator <energy> $E=(\dot x^2+x^2)/2$. The <differential equation> gives
$$
\dot E=-\varepsilon(2\dot x^4+2x^3\dot x-4x^4\dot x^2-\dot x^2).
$$
For a leading-order orbit $x=A\cos\theta$, $\dot x=-A\sin\theta$, the cycle averages are $\langle\dot x^4\rangle=3A^4/8$, $\langle x^3\dot x\rangle=0$, $\langle x^4\dot x^2\rangle=A^6/16$, and $\langle\dot x^2\rangle=A^2/2$. The <energy balance method> therefore gives
$$
\langle\dot E\rangle=\frac\varepsilon4A^2(A^2-1)(A^2-2),\qquad\dot A=\frac\varepsilon4A(A^2-1)(A^2-2).
$$
The two nonzero simple roots give the leading <limit cycles> \b[$x\simeq\cos(t+\theta_0)$ and $x\simeq\sqrt2\cos(t+\theta_0)$]. Below amplitude $1$ the <energy> increases, between $1$ and $\sqrt2$ it decreases, and above $\sqrt2$ it increases. Hence \b[the amplitude-$1$ cycle is stable and the amplitude-$\sqrt2$ cycle unstable] for positive small $\varepsilon$.

The linearized averaged radial exponents are $-\varepsilon/2$ and $\varepsilon$, respectively, consistent with nontrivial <Floquet multipliers> $e^{-\pi\varepsilon+O(\varepsilon^2)}$ and $e^{2\pi\varepsilon+O(\varepsilon^2)}$. The nonzero derivatives at these averaged zeros also justify persistence of nearby cycles for sufficiently small $\varepsilon$.