Solution (source code)

= Solution

A <Lyapunov function> near an equilibrium is a continuously differentiable positive-definite function $V$, vanishing there, whose <derivative> along trajectories is nonpositive. The strict <Second Lyapunov theorem> says a negative-definite <derivative> implies local asymptotic stability.

For sharper bounds than the circular test $x^2+y^2$, put $z=(x,y)^T$, $r^2=z^Tz$ and
$$
A=\begin{pmatrix}-1&1\\-2&-1\end{pmatrix},\quad
P=\begin{pmatrix}4/3&-1/6\\-1/6&5/6\end{pmatrix},\quad V=z^TPz.
$$
A direct multiplication gives $A^TP+PA=-2I$. Its <eigenvalues> are $\lambda_\pm=(13\pm\sqrt{13})/12>0$. Since the nonlinear vector field is $Az+r^2z$,
$$
\boxed{\dot V=2r^2(V-1)}.
$$
For $0<V<1$, the <derivative> is strictly negative. The Lyapunov theorem proves asymptotic stability. In fact $V(0)<1$ gives $V(t)\leq V(0)$ and
$$
\dot V\leq-\frac{2(1-V(0))}{\lambda_+}V,
$$
so the solution stays bounded, exists for all positive time and converges exponentially to zero.

The ellipse $V=1$ is invariant. In polar coordinates, $\dot\theta=-(2\cos^2\theta+\sin^2\theta)<0$, so it is a nonconstant periodic orbit, not part of the basin. For $V(0)>1$, use $r^2\geq V/\lambda_+$ to obtain $\dot V\geq2V(V-1)/\lambda_+$. This implies finite-time blow-up by comparison with the scalar equation. Thus the exact basin is $\{V<1\}$ and every point outside its closure escapes to unbounded norm within its maximal existence interval.

The largest open disc centred at zero lying in this exact basin has radius
$$
\boxed{r_1=\lambda_+^{-1/2}=\sqrt{\frac{12}{13+\sqrt{13}}}}.
$$
The smallest radius such that every point strictly outside its circle escapes is the largest radius on the invariant ellipse:
$$
\boxed{r_2=\lambda_-^{-1/2}=\sqrt{\frac{12}{13-\sqrt{13}}}}.
$$
These are optimal: a larger attraction disc contains a periodic-orbit point, and a smaller escape threshold leaves a periodic-orbit point in its exterior. This is an <exact quadratic basin boundary>.

The simpler radial <derivative> is $\tfrac12(d/dt)r^2=r^2(r^2-1)-xy$. It only certifies attraction for $r<1/\sqrt2$ and strictly increasing norm for $r>\sqrt{3/2}$. Those are optimal thresholds for a sign test on that radial <derivative>, not for the actual attraction and escape regions. If “increases without bound” is read as requiring monotonic increase from the initial instant, the smallest such uniform radius is $\sqrt{3/2}$: at direction $x=y$, any smaller threshold permits an initially decreasing norm. The sharper $r_2$ above concerns eventual unbounded escape, which need not be monotone at every instant.