The region is compact and positively invariant and contains the unique fixed point. If that fixed point is unstable, choose an initial point in which is not on its stable set. Its nonempty compact -limit set cannot be the fixed point. The Poincaré-Bendixson theorem therefore supplies a periodic orbit. Hence a sufficient parameter condition, and the one selected by the preceding analysis, is
For , . It points outward on a sufficiently small circle and inward on a circle . The intervening annulus is compact, forward invariant, and contains no equilibrium. Poincaré-Bendixson theorem therefore gives a periodic orbit.
The non-axis nullclines are and . For , , the first has intercepts and the second has . Their axis intercept order rules out a positive intersection. Below the first line increases and above it decreases; below the second increases and above it decreases. Thus is a saddle point and is stable.
Figure 1.
Competition nullclines and trajectories as grey-squirrel removal increases
. Original dimensionless model plots with , , and . The three panels illustrate grey dominance, bistability, and red dominance. Solid lines are the red and grey non-axis nullclines; arrows and trajectories show the direction of motion. Details.
For strictly positive initial populations, both coordinates stay positive and are bounded above by their individual logistic equations. There are no interior periodic orbits: the Dulac function makes the divergence of the multiplied vector field , so the Bendixson-Dulac theorem applies. The stable manifold of is the invariant red-only axis, and is repelling. The phase portrait, or the Poincaré-Bendixson theorem with these boundary directions, therefore gives
The qualification matters: a grey population identically absent cannot drive extinction. In dimensional variables, positive grey presence leads to , in this no-hunting case.
At , the origin has stable and unstable eigenvalues and ; the two nonzero equilibria are repelling foci with eigenvalues . A compatible portrait has one attracting closed orbit surrounding all three equilibria. Its total enclosed index is , as a periodic orbit requires. Its half-turn image is also attracting, so uniqueness of the attracting cycle makes that cycle half-turn symmetric. Orbits spiral away from either focus equilibrium and approach the surrounding cycle; the stable separatrices approach the saddle instead. The figure integrates representative trajectories and shows the local saddle directions.
These trajectories cannot escape to infinity. A Liénard boundedness function supplies one explicit boundedness argument using the Liénard coordinate with and . Then and . Put and . This function is coercive after addition of a constant. Its derivative obeys
using . The right side is negative outside a compact region, yielding bounded forward motion.
In the displayed portrait, generic non-equilibrium initial states tend to the cycle; points on the saddle's stable manifold tend to the origin; and initial data exactly at a repelling focus equilibrium remain there. The Poincaré-Bendixson theorem excludes a chaotic compact limit set for this smooth planar flow. Strictly, the assumption gives uniqueness only of an attracting periodic orbit: it does not by itself exclude exceptional nonattracting periodic or separatrix limit sets. Those would need additional global information if one wanted a classification stronger than the compatible portrait requested.
Figure 1.
A compatible attracting cycle, repelling foci and saddle separatrices at Omega minus two
. Details.
The Poincaré-Bendixson theorem states that the omega-limit set of a forward orbit confined to a compact region of the plane, containing no equilibrium points, is a periodic orbit.
Put , . Direct calculation gives
Here and . Thus on the inner circle the radial velocity is positive, and on the outer circle it is negative. In their closed annulus,
so there is no equilibrium. The theorem gives a periodic orbit in this trapping annulus.
It is asymptotically stable: the vector-field divergence is
throughout the annulus. Its nontrivial Floquet multiplier is . This proves an attracting periodic orbit exists. There can be only one there: two such nested orbits would enclose an annular region with zero outward flux, contradicting the strictly negative divergence via the divergence theorem.
A positive equilibrium must have . Let
It is the unique positive root of . At this point the Jacobian matrix is
Consequently
It is linearly asymptotically stable when , and unstable when . In either region it is a node or focus according as is nonnegative or negative. At the linear eigenvalues are purely imaginary. If , it is stable for every .
The requested boundary follows from and :
To prove a limit cycle exists in the unstable case, a compact trapping set must stay away from the singular boundary . The following construction supplies it. Choose large enough to contain the positive equilibrium and the orbit being considered. On , , and on with , . Put . Using and completing the square in gives
Choose so small that times the square bracket is less than , and then choose . The set
is compact in the positive quadrant and forward invariant. On the curved boundary , and . On , gives . Its other boundaries were checked above. The constants can be chosen so the equilibrium is interior.
When , the equilibrium is a hyperbolic repeller. Remove a sufficiently small outward-flow neighborhood of it from . This leaves a compact trapping region containing no equilibria. The Poincaré-Bendixson theorem gives a periodic omega-limit orbit of a nonequilibrium trajectory, hence a limit cycle. The analytic vector field excludes an accumulating family of periodic return-map zeros unless the whole local return map is the identity, which is incompatible with the repelling equilibrium and the nonperiodic approaching trajectory. This is the positive trapping construction for the Leslie-Gower model.
A rectangular upper bound , , is inward pointing for the Leslie-Gower predator-prey model. To avoid the singular edge , use and the inequality , obtained by completing a square. Choose sufficiently small and . Then , , is a compact forward-invariant set in the positive quadrant. Removing an outward-flow neighborhood of a repelling equilibrium leaves a trapping region without equilibria, so the Poincaré-Bendixson theorem yields a periodic omega-limit orbit.