Solution (source code)

= Solution
{c}

Write $\eta=H e^{\sigma t}J_0(\alpha r)$ using the <Bessel function of the first kind>. The wall <boundary conditions> at $z=\pm h$ select
$$
\phi_1=C_1\cosh[\alpha(z+h)]J_0(\alpha r)e^{\sigma t},\qquad
\phi_2=C_2\cosh[\alpha(h-z)]J_0(\alpha r)e^{\sigma t}.
$$
Because $J_0'(s)=-J_1(s)$, the sidewall <no-penetration boundary condition> requires
$$
\boxed{J_1(\alpha_na)=0,\qquad \alpha_n=\frac{j_{1,n}}a,\quad n=1,2,\ldots,}
$$
where $j_{1,n}$ are the positive zeros of $J_1$. These radial <eigenfunctions> automatically preserve layer volume because
$$
\int_0^a rJ_0(\alpha_nr)\,dr=\frac{aJ_1(\alpha_na)}{\alpha_n}=0.
$$
The constant radial mode $\alpha=0$ cannot change the interface height in a closed tank: the <Laplace equation> and the impermeable lids then give zero vertical <velocity>, and a uniform displacement would change each layer's volume.

The <kinematic boundary condition> fixes
$$
C_1=\frac{\sigma H}{\alpha\sinh(\alpha h)},\qquad
C_2=-\frac{\sigma H}{\alpha\sinh(\alpha h)}.
$$
Thus the interface values are $\phi_1(0)=\sigma\eta\coth(\alpha h)/\alpha$ and $\phi_2(0)=-\sigma\eta\coth(\alpha h)/\alpha$. Substitution in the <normal stress> condition gives the <finite-depth Rayleigh-Taylor dispersion relation>:
$$
\frac{(\rho_1+\rho_2)\sigma_n^2}{\alpha_n}\coth(\alpha_nh)
=(\rho_2-\rho_1)g,
$$
so
$$
\boxed{\sigma_{n,\pm}=\pm\sqrt{\mathcal A g\alpha_n\tanh(\alpha_nh)},\qquad
\mathcal A=\frac{\rho_2-\rho_1}{\rho_2+\rho_1}>0.}
$$
Here $\mathcal A$ is the <Atwood number>. Every allowed mode has a positive <growth rate> and is subject to <Rayleigh-Taylor instability>. Since $d[\alpha\tanh(\alpha h)]/d\alpha=\tanh(\alpha h)+\alpha h\operatorname{sech}^2(\alpha h)>0$, shorter radial scales grow faster. The limits are
$$
\sigma_{n,+}\sim\sqrt{\mathcal A gh}\,\alpha_n\quad(\alpha_nh\ll1),\qquad
\sigma_{n,+}\sim\sqrt{\mathcal A g\alpha_n}\quad(\alpha_nh\gg1).
$$
There is no fastest finite <wavenumber> in the ideal sharp-interface model without <surface tension>, <viscosity> or <diffusion>. The zero-initial-<velocity> condition uses both exponential branches: an initial mode $H J_0(\alpha_nr)$ evolves as $H\cosh(\sigma_{n,+}t)J_0(\alpha_nr)$. If the <mass density> ordering were reversed the same calculation would give oscillatory stable modes; equal <mass densities> give neutral modes.