Solution (source code)

= Solution

For small irrotational <surface gravity waves>, write the <velocity field> as $\nabla\phi$. <Incompressibility> gives $\nabla^2\phi=0$. The rigid walls have $\phi_x=0$ at $x=0,a$, $\phi_y=0$ at $y=0,b$, and $\phi_z=0$ at $z=-h$. At $z=0$, the linear <kinematic boundary condition> and pressure condition are
$$
\eta_t=\phi_z,\qquad \phi_t+g\eta=0.
$$
Separate variables for a <rectangular standing surface-gravity mode>:
$$
\phi=B(t)\cos\frac{m\pi x}{a}\cos\frac{n\pi y}{b}
\frac{\cosh k(z+h)}{\cosh kh},\qquad
k^2=\pi^2\left(\frac{m^2}{a^2}+\frac{n^2}{b^2}\right).
$$
The horizontal cosine factors and vertical hyperbolic cosine satisfy all rigid-wall conditions. Put $\eta=A(t)\cos(m\pi x/a)\cos(n\pi y/b)$ and $\lambda=k\tanh kh$. The free-surface conditions give $\dot A=\lambda B$ and $\dot B=-gA$, hence
$$
\boxed{\omega^2=gk\tanh kh.}
$$
Both integers are nonnegative and not both zero. The zero pair would be a spatially uniform displacement, excluded by fixed fluid volume; a spatially constant potential is only a gauge mode.