 Solution

Put $z=e^{i\theta }$, so $d\theta =dz/(iz)$, $\cos 2\theta =(z^2+z^{-2})/2$ and ${\cos }^{2}3\theta =(z^6+2+z^{-6})/4$. The contour integral isThe enclosed poles are $0$ and $\pm 1/\sqrt{2}$. For the rational function displayed without the prefactor $1/(4i)$, the residue at zero is $-21/8$ and the two other residues sum to $27/8$. Therefore the residue theorem givesA useful independent check is the Fourier series $(5-4\cos \varphi {)}^{-1}=\frac{1}{3}[1+2{\sum }_{k\ge 1}{2}^{-k}\cos k\varphi ]$; only its constant and third Fourier coefficients survive.