Solution (source code)

= Solution

Use the complete system in the PDF: the TeX omits $\dot y=y(4x-1)/8$. Both coordinate axes are invariant, and the <equilibrium points> are $(0,0)$, $(1,0)$ and $(1/4,3/4)$. The <Jacobian matrix> is
$$
J(x,y)=\begin{pmatrix}1-2x-y&-x\\y/2&(4x-1)/8\end{pmatrix}.
$$
At $(0,0)$ its <eigenvalues> are $1,-1/8$, and at $(1,0)$ they are $-1,3/8$: both points are <saddle equilibria>. At the coexistence point,
$$
J_* =\begin{pmatrix}-1/4&-1/4\\3/8&0\end{pmatrix},\qquad
\boxed{\lambda_\pm=\frac{-1\pm i\sqrt5}{8}.}
$$
Thus the interior point is a <stable spiral>. The nontrivial <nullclines> are $x+y=1$ and $x=1/4$: $x$ increases below the first and decreases above it, while $y$ increases to the right of the second and decreases to the left. The rotation near coexistence is counterclockwise.

\Image[/past-exam-of-the-mathematics-course-of-the-university-of-cambridge/2017/ia/paper-2-predator-prey.png]
{title=Trajectories of a predator-prey system}

A local <linearization> by itself does not prove that every interior solution has the same limit. For this <logistic predator-prey model>, let $x_*=1/4,y_*=3/4$ and use the <Lyapunov function>
$$
V=x-x_*-x_*\log(x/x_*)
+2\bigl[y-y_*-y_*\log(y/y_*)\bigr].
$$
The inequality $u-1-\log u\geq0$ makes $V$ nonnegative, with equality only at coexistence. Its sublevel sets are compact inside the positive quadrant, because $V$ diverges both at the axes and at infinity. Direct differentiation cancels the predator-prey cross terms and gives
$$
\dot V=-(x-x_*)^2\leq0.
$$
On $\{\dot V=0\}$ one has $x=x_*$, and remaining there requires $\dot x=-x_*(y-y_*)=0$. The largest invariant subset is therefore the coexistence point. The <LaSalle invariance principle> proves
$$
\boxed{(x(t),y(t))\longrightarrow(1/4,3/4)
\quad\text{for every }x(0)>0,\ y(0)>0.}
$$
On $x=0$, $y$ decays as $e^{-t/8}$ toward the origin. On $y=0$, the <logistic differential equation> takes every $x(0)>0$ to one. The origin itself stays fixed. These boundary trajectories are the exceptions to the interior convergence statement.