 Solution

For the self-gravitating incompressible slab, let the fluid occupy $-H<z<H$, with mass density ${\rho }_0=\Sigma /(2H)$ and a vacuum exterior. Poisson equation and hydrostatic equilibrium giveA constant ambient pressure could be added without changing the result. Choose perturbations proportional to $e^{ikx-i\omega t}$, using rotational symmetry in the horizontal plane to put the wavevector along $x$.

For the fluid displacement, the linearized momentum equation inside the uniform slab is $-{\omega }^2\mathbf{\xi }=-\nabla (p^{\prime }/{\rho }_0+{\varphi }^{\prime })$. For $\omega \ne 0$ it follows that $\mathbf{\xi }=\nabla \chi$, with $p^{\prime }/{\rho }_0+{\varphi }^{\prime }={\omega }^2\chi$. Incompressibility gives $\nabla \cdot \mathbf{\xi }=0$, henceNeutral limits of these surface modes follow by continuity. Stationary vorticity perturbations form a separate zero-frequency sector; the nonzero-frequency surface modes are irrotational by the momentum equation itself, without an extra irrotational-flow assumption.

The Eulerian mass density perturbation vanishes in the bulk because the equilibrium mass density is uniform and the displacement is divergence free. If the two vertical surface displacements are ${\eta }_{+}={\xi }_z(H)$ and ${\eta }_{-}={\xi }_z(-H)$, the surface density perturbation of a displaced uniform interface givesThus Poisson equation reduces to Laplace equation for ${\varphi }^{\prime }$ inside and outside. The potential is continuous, while integrating through each surface givesThe perturbing potential must decay at vertical infinity. The kinematic conditions are ${\eta }_{\pm }={\partial }_z\chi (\pm H)$. The free surface pressure condition is the vanishing Lagrangian pressure perturbation, $p^{\prime }+{\eta }_{\pm }{p}_0^{\prime }=0$; at the top this gives $p^{\prime }(H)/{\rho }_0=g_s{\eta }_{+}$.

Put $q=kH>0$. For even scalar-potential symmetry, chooseThen ${\eta }_{-}=-{\eta }_{+}$: this is a thickness-changing mode, not a vertical bending mode. Write $\eta ={\eta }_{+}$; the top derivative jump gives $-kC(\cosh q+\sinh q)=4\pi G{\rho }_0\eta$. Since $\cosh q+\sinh q=e^q$,Substitute these values into ${\omega }^2\chi (H)=g_s\eta +{\varphi }^{\prime }(H)$. Using $4\pi G{\rho }_0=g_s/H$ gives the even-mode dispersion relation for the surface modes of a self-gravitating incompressible slab:The positive term comes from the restoring hydrostatic free-surface pressure in the background gravitational field. The negative term is the attraction due to the perturbed gravitational potential of the displaced surfaces, which tends to reinforce the disturbance. Both effects ultimately involve the slab's gravity, but enter the boundary condition through different terms.

For odd scalar-potential symmetry, instead takeNow ${\eta }_{-}={\eta }_{+}$, so the surfaces bend in the same direction. The jump again gives $C=-4\pi G{\rho }_0\eta /(k{e}^q)$, but $\chi (H)=\eta \tanh q/k$ and ${\varphi }^{\prime }(H)=-4\pi G{\rho }_0\eta /[k(1+\coth q)]$. The same pressure condition yieldsThe bottom kinematic, jump and pressure conditions follow with the indicated parity and give the same dispersion relation, so no boundary condition has been discarded.

To determine the gravitational instability of an incompressible slab, factor the two dimensionless expressions asFor the even mode the bracket is strictly increasing, with derivative $1+e^{-2q}>0$, is negative at zero and tends to positive infinity. It therefore has a unique positive zeroThe even mode is unstable for $0<kH<q_c$, neutral at $q_c$, and oscillatory for $kH>q_c$. Negative ${\omega }_e^2$ gives a growing mode with $\omega =i\sqrt{-{\omega }_e^2}$ under the stated time convention. For the odd mode, $1-e^{-2q}<2q$ for every $q>0$, so the bracket is always positive: all odd modes with positive wavenumber are stable.

At long wavelengths $F_e=-q+2q^2+O(q^3)$ and $F_o=q-2q^2/3+O(q^3)$: a thickness or column-density disturbance has a gravitational instability, whereas a bending disturbance is restored. At short wavelengths both approach $q-1/2$, so both surface branches are stable. The slab equilibrium is therefore linearly unstable overall, because it admits the long-wavelength even modes despite the stability of its odd sector.