 Solution

Put $A=(Gm/h^3)F(hk)$. The two amplitude equations have coefficient matrixIts determinant gives the dispersion relationWriting $q=A/{\Omega }^2\ge 0$, its squared growth rates areand the four growth rates are both square roots of each displayed value. DefineFor $0<q<q_{-}$, the discriminant is positive, the sum of the squared rates is $q-1<0$, and their product is $2q(3-q)>0$; both squared rates are negative. These modes have purely imaginary $s$. For $q_{-}<q<q_{+}$, the squared rates form a nonreal complex conjugate pair; their square roots include roots with positive real part. This is overstability, meaning growing oscillations. For $q\ge q_{+}$, at least the plus squared rate is positive because $q>1$, so a real growing root exists. Consequently every $q>q_{-}$ is exponentially unstable.

Take the maximizing mode $hk=\pi$, so $q=(Gm/(h^3{\Omega }^2))F(\pi )$. It is unstable wheneverThis is also the threshold for the onset of exponential growth among all lattice modes, since $0\le F\le F(\pi )$. At equality the maximizing mode has repeated imaginary roots and can have algebraic growth; it is a marginal collision of oscillatory modes, not a strict-decay assertion. The $k=0$ mode has $F=0$, an epicyclic motion sector and a zero-frequency epicyclic guiding center sector with possible secular shear drift. These neutral or algebraic behaviors are distinct from the strict exponential instability established by the requested inequality.