Solution (source code)

= Solution

The background has $\nabla a_0=0$, so the linear chemotaxis term is $-\chi_0n_0\nabla^2\xi$; derivatives of $\chi$ multiply products of disturbances and do not enter at first order. For a <normal mode> with $\nabla^2=-k^2$ and time dependence $e^{pt}$, put $q=k^2$. Then
$$
p\begin{pmatrix}\eta\\\xi\end{pmatrix}=\begin{pmatrix}-A-q&\chi_0n_0q\\\beta&-\gamma-Dq\end{pmatrix}\begin{pmatrix}\eta\\\xi\end{pmatrix},
$$
and the <characteristic polynomial> is
$$
\boxed{p^2+[A+\gamma+(1+D)q]p+Dq^2+(AD+\gamma-\beta\chi_0n_0)q+A\gamma=0.}
$$
Its linear coefficient is positive for every $q\geq0$. Thus instability occurs exactly when its constant term $Q(q)$ is negative: the roots are then real with opposite signs. The positive quadratic $Q$ is negative somewhere on $q>0$ precisely when
$$
\beta\chi_0n_0-AD-\gamma>2\sqrt{DA\gamma}.
$$
Therefore
$$
\boxed{\beta\chi_0n_0>2\alpha n_0^2D+\gamma+2\sqrt{2D\alpha n_0^2\gamma}.}
$$
At equality the neutral wavenumber is $q_c=\sqrt{A\gamma/D}$ and $p=0$. Above threshold the unstable band is $q_-<k^2<q_+$, where
$$
q_\pm=\frac{B\pm\sqrt{B^2-4DA\gamma}}{2D},\qquad B=\beta\chi_0n_0-AD-\gamma.
$$