Solution (source code)

= Solution

The transition rates of this <birth-death process> are $q_{n,n+1}=\lambda n$ and $q_{n,n-1}=n(\gamma+\beta n)$. Thus, with $p_{-1}=0$,
$$
\dot p_n=\lambda(n-1)p_{n-1}+(n+1)(\gamma+\beta(n+1))p_{n+1}
-[\lambda n+n(\gamma+\beta n)]p_n.
$$
Multiplying by $n$, summing, and shifting indices gives
$$
\frac d{dt}\mathbb E N=(\lambda-\gamma)\mathbb EN-\beta\mathbb E(N^2).
$$
At a finite-moment steady state with positive mean and $\lambda>\gamma$,
$$
\beta\mathbb E(N^2)=(\lambda-\gamma)\mathbb EN.
$$
Since $\mathbb E(N^2)\geq(\mathbb EN)^2$, this implies
$$
0<\mathbb EN\leq\frac{\lambda-\gamma}{\beta}.
$$
If $\lambda\leq\gamma$, both terms in the mean equation are nonpositive, and stationarity forces $\mathbb EN=0$. Thus the population is extinct almost surely in any such steady state.