Solution (source code)

= Solution

Put $r^2=x^2+y^2$. The system can be written
$$
\begin{pmatrix}\dot x\\\dot y\end{pmatrix}
=
\begin{pmatrix}\mu&-1\\1&\mu\end{pmatrix}
\begin{pmatrix}x\\y\end{pmatrix}
-r^2(r^2-\lambda)\begin{pmatrix}x\\y\end{pmatrix}.
$$
It therefore vanishes at the origin. The linearization there has eigenvalues
$$
\mu\pm i.
$$
Hence the origin is an asymptotically stable focus for $\mu<0$ and an unstable focus for $\mu>0$.

When $\mu=0$ the linearization alone is inconclusive. Direct radial reduction gives
$$
\dot r=r^3(\lambda-r^2),
$$
which is positive for every sufficiently small $r>0$. The origin is therefore unstable at $\mu=0$ as well, although trajectories leave it only at cubic order.