Solution (source code)

= Solution

The equilibrium equations are $x(-1+x^2+\beta y^2)=y(-1+\beta x^2+y^2)=0$. For $\beta\ne1$ the <fixed points> are the origin, four axis points $(\pm1,0),(0,\pm1)$, and four diagonal points $(\pm s,\pm s)$ with independent signs and $s=(1+\beta)^{-1/2}$. The hypothesis $\beta>-1$ ensures that $s$ is real. The <Jacobian matrix> has diagonal entries $-1+3x^2+\beta y^2$, $-1+\beta x^2+3y^2$ and off-diagonal entries $2\beta xy$.

At the origin both <eigenvalues> are $-1$, so it is a <stable node>. At each axis point the <eigenvalues> are $2,\beta-1$. At each diagonal point they are $2,2(1-\beta)/(1+\beta)$, along the radial and transverse diagonal directions respectively. Thus
$$
\boxed{\begin{array}{c|cc}
&-1<\beta<1&\beta>1\\\hline
\text{origin}&\text{sink}&\text{sink}\\
\text{axis points}&\text{saddles}&\text{sources}\\
\text{diagonal points}&\text{sources}&\text{saddles}
\end{array}}.
$$
The axes and diagonals are invariant lines. For $\beta=1/2$ the axis <saddle points> have stable directions transverse to their axes and unstable directions along them; the diagonal points are <unstable nodes>. For $\beta=2$ these roles switch: the diagonal <saddle points> have stable transverse directions and unstable radial directions, and the axis points are <unstable nodes>. Their <stable manifolds> separate trajectories attracted to the origin from outward-escaping trajectories. Symmetry gives the other quadrants.

For $\beta=1$, the entire unit circle consists of equilibria. In <polar coordinates> $\dot r=r(r^2-1)$ and $\dot\theta=0$: every ray inside the circle moves toward the origin, and every ray outside moves outward. The circle has neutral tangential and unstable radial directions; its points are not isolated <saddle points> or <unstable nodes>.

All three <phase portraits> are shown below. The potential $\Phi=-(x^2+y^2)/2+(x^4+y^4)/4+\beta x^2y^2/2$ satisfies $(\dot x,\dot y)=\nabla\Phi$ and $\dot\Phi=|\nabla\Phi|^2$, ruling out nonconstant periodic orbits. Blue curves show the saddle stable <separatrices>, integrated backwards from their local stable directions. The flow arrows and invariant lines, together with the equilibrium classification, specify the topology of the sketches.

\Image[/past-exam-of-the-mathematics-course-of-the-university-of-cambridge/2017/ii/paper-1-phase-portraits.png]
{title=<Phase portraits> and equilibrium branches}