Solution (source code)

= Solution

The top and bottom fluxes cancel because $F_z=x^2+y^2$ is independent of $z$. Parametrise the side by $(\cos\theta/\sqrt a,\sin\theta/\sqrt b,z)$; its outward <vector> area is
$$
\left(\frac{\cos\theta}{\sqrt b},\frac{\sin\theta}{\sqrt a},0\right)d\theta\,dz.
$$
The term involving $y^2z$ integrates to zero, and the remaining <integral> is
$$
\int_0^3\int_0^{2\pi}\frac{3z\cos^2\theta}{\sqrt{ab}}\,d\theta\,dz
=\frac{27\pi}{2\sqrt{ab}},
$$
confirming part (a).