Solution (source code)

= Solution

The <fixed points> are $(0,0)$ and $(\pm|\mu|,\mu^2)$. The <Jacobian matrix> is
$$
J(x,y)=\begin{pmatrix}\mu^2-y&-x\\2x&-1\end{pmatrix}.
$$
At the origin its <eigenvalues> are $\mu^2$ and $-1$, so it is a <saddle equilibrium> for every allowed $\mu$. At either nonzero fixed point the <characteristic polynomial> is $\lambda^2+\lambda+2\mu^2$, giving
$$
\boxed{\lambda_\pm=\frac{-1\pm\sqrt{1-8\mu^2}}2.}
$$
Thus both are <stable nodes> for $0<|\mu|<1/(2\sqrt2)$ and <stable focuses> for $|\mu|>1/(2\sqrt2)$. At equality the repeated <eigenvalue> is $-1/2$ and the <matrix> is not scalar, so there is only one independent <eigenvector>: the fixed point is a stable improper node.

The <stable manifold> of the origin is \b[$x=0$], exactly: there $\dot y=-y$. Conversely, a trajectory converging to the origin with $x\ne0$ would eventually have $\dot x/x=\mu^2-y>\mu^2/2$, which precludes convergence of $x$ to zero.

For the local <unstable manifold>, write $y=h(x)=ax^2+bx^4+O(x^6)$. Reflection symmetry makes $h$ even, and the <invariance equation for a graph> requires
$$
h'(x)x(\mu^2-h(x))=-h(x)+x^2.
$$
Comparing quadratic and quartic terms gives $2a\mu^2=1-a$ and $4b\mu^2-2a^2=-b$. Therefore
$$
\boxed{y=\frac{x^2}{1+2\mu^2}+\frac{2x^4}{(1+4\mu^2)(1+2\mu^2)^2}+O(x^6).}
$$