Solution (source code)

= Solution

At $\varepsilon=0$,
$$
H(x,y)=2x^2+\frac12y^2
$$
is conserved, and its orbit of amplitude $a>0$ may be written
$$
x=a\cos\theta,\qquad y=-2a\sin\theta,\qquad \theta=2t.
$$
The energy change over one period, to first order in $\varepsilon$, is
$$
\begin{aligned}
\Delta H
&=\varepsilon\int_0^\pi
\left[k(1-2x^2)y^2-(1-3x^2)y^4\right]dt+O(\varepsilon^2)\\
&=\varepsilon\pi a^2\left[2k-(k+6)a^2+3a^4\right]+O(\varepsilon^2)\\
&=3\varepsilon\pi a^2(a^2-2)(a^2-k/3)+O(\varepsilon^2).
\end{aligned}
$$
Here the stated trigonometric integrals, together with
$\int_0^{2\pi}\cos^2\theta\sin^4\theta\,d\theta=\pi/8$, were used. The two candidate leading-order amplitudes are therefore
$$
\boxed{a=\sqrt2,\qquad a=\sqrt{k/3}}.
$$