 Solution

For $h=x^2/a^2+y^2/b^2+z^2-1$, the gradient $\nabla h=2(x/a^2,y/b^2,z)$ is a normal vector to the ellipsoid. The outward unit normal vector is $(x/a^2,y/b^2,z)/\sqrt{x^2/a^4+y^2/b^4+z^2}$. The conditions $a,b>\sqrt{2}$ ensure that $z=\sqrt{1-x^2/a^2-y^2/b^2}>0$ everywhere on the rectangular projection.

The dot product $F\cdot \nabla h=2xy(1/b^2-1/a^2)$ vanishes pointwise when $a=b$. In that case $F$ has no normal component, so the surface flux is zero. For unequal axes, the given vector surface element givesPut $u=x^2/a^2$, $v=y^2/b^2$. Then $xydxdy=a^2{b}^{2}dudv/4$, andTwo elementary integrations now giveThis is the rotational flux through an ellipsoidal graph patch with unit vertical semiaxis and unit projection lengths. It also tends continuously to zero when $a=b$, agreeing with the pointwise argument.