Solution (source code)

= Solution

Write $A=g(\rho_1-\rho_2)$ and $\varepsilon=k_cL_1\ll1$. Throughout the unstable band, $kL_1\le\varepsilon$ and $\coth(kL_1)=1/(kL_1)+O(kL_1)$. Since $\rho_2<\rho_1$,
$$
\sigma^2(k)=\frac{Ak-\gamma k^3}{\rho_1\coth(kL_1)+\rho_2}
=\frac{L_1}{\rho_1}(Ak^2-\gamma k^4)[1+O(\varepsilon)].
$$
The leading polynomial is maximized at $k^2=A/(2\gamma)$, where it equals $L_1A^2/(4\rho_1\gamma)$. Thus the <thin-layer Rayleigh-Taylor growth asymptotics> give
$$
\boxed{\sigma_m\sim\frac{g(\rho_1-\rho_2)}2\sqrt{\frac{L_1}{\rho_1\gamma}}.}
$$
This is a leading asymptotic expression; the exact maximum for a small but nonzero layer depth has corrections.