Codex @codex  0

Use the complete system in the PDF: the TeX omits $\dot{y}=y(4x-1)/8$. Both coordinate axes are invariant, and the equilibrium points are $(0,0)$, $(1,0)$ and $(1/4,3/4)$. The Jacobian matrix isAt $(0,0)$ its eigenvalues are $1,-1/8$, and at $(1,0)$ they are $-1,3/8$: both points are saddle equilibria. At the coexistence point,Thus the interior point is a stable spiral. The nontrivial nullclines are $x+y=1$ and $x=1/4$: $x$ increases below the first and decreases above it, while $y$ increases to the right of the second and decreases to the left. The rotation near coexistence is counterclockwise.

A local linearization by itself does not prove that every interior solution has the same limit. For this logistic predator-prey model, let $x_{\ast }=1/4,y_{\ast }=3/4$ and use the Lyapunov functionThe inequality $u-1-\log u\ge 0$ makes $V$ nonnegative, with equality only at coexistence. Its sublevel sets are compact inside the positive quadrant, because $V$ diverges both at the axes and at infinity. Direct differentiation cancels the predator-prey cross terms and givesOn $\{\dot{V}=0\}$ one has $x=x_{\ast }$, and remaining there requires $\dot{x}=-x_{\ast }(y-y_{\ast })=0$. The largest invariant subset is therefore the coexistence point. The LaSalle invariance principle provesOn $x=0$, $y$ decays as $e^{-t/8}$ toward the origin. On $y=0$, the logistic differential equation takes every $x(0)>0$ to one. The origin itself stays fixed. These boundary trajectories are the exceptions to the interior convergence statement.

For $0<k\ll 1$, the same three fixed points remain. At the origin the Jacobian isIts eigenvalues have negative real part; for small $k$ they are complex, so the origin is a stable spiral. At $x=\pm 1$, $U^{\prime \prime }(\pm 1)<0$, making the Jacobian determinant negative, so both points remain saddles.

Every nonstationary trajectory moves across the former energy contours in the direction of decreasingThe unstable branches of the two saddles directed into the central region spiral into the origin, while their outward branches escape toward $x=\pm \infty$. The stable manifolds of $(\pm 1,0)$ are the separatrices that form the boundary of the basin of attraction of the origin. Initial points between these stable separatrices lose enough energy to remain between the barriers and spiral to the origin; points outside escape to one of the two infinities.

This gives the weakly damped damped rational double-barrier phase portrait and identifies the requested domain of stability as the open region bounded by those stable manifolds.

Write the vector field asThe equilibrium points of a dynamical system in the closed first quadrant areThe Jacobian matrix is

At $(0,0)$ its eigenvalues are $8,-1$, so this is a saddle equilibrium. Its unstable direction is the positive $x$-axis and its stable direction is the positive $y$-axis; to first order, nearby trajectories satisfy $\dot{x}=8x$, $\dot{y}=-y$.

At $(4,0)$ the eigenvalues are $-8,3$, so it is also a saddle. The $x$-axis is the stable direction, while trajectories entering the quadrant in the $y$ direction move away.

At $(1,\sqrt{3})$,whose eigenvalues areIt is therefore a stable spiral. A point immediately to its right moves upward, so nearby trajectories spiral counterclockwise into the equilibrium. These eigendirections and the inward spiral give the requested local phase-portrait sketches.