 Solution

Put $q=k_n^2$. The mode matrix has trace $-1/2-(1+D)q<0$, so a growing eigenvalue occurs exactly when its determinant is negative:This upward quadratic has two positive roots only if $D<1/2$ and $D^2-5D+1/4>0$. Within $0<D<1$ this isFor such $D$, defineThe unstable modes satisfy $\boxed{\sqrt{q_{-}}<k_n<\sqrt{q_{+}}}$. These are strict inequalities: at either endpoint the leading eigenvalue is zero rather than positive.