Solution (source code)

= Solution

Put $z=e^{i\theta}$, so $d\theta=dz/(iz)$, $\cos2\theta=(z^2+z^{-2})/2$ and $\cos^23\theta=(z^6+2+z^{-6})/4$. The <contour integral> is
$$
\frac1{4i}\oint_{|z|=1}\frac{z^8+2z^2+z^{-4}}{z(5z^2-2z^4-2)}\,dz.
$$
The enclosed <poles> are $0$ and $\pm1/\sqrt2$. For the rational function displayed without the prefactor $1/(4i)$, the <residue> at zero is $-21/8$ and the two other <residues> sum to $27/8$. Therefore the <residue theorem> gives
$$
\boxed{\int_0^{2\pi}\frac{\cos^23\theta}{5-4\cos2\theta}\,d\theta=\frac1{4i}(2\pi i)\frac34=\frac{3\pi}{8}.}
$$
A useful independent check is the <Fourier series> $(5-4\cos\phi)^{-1}=\frac13[1+2\sum_{k\ge1}2^{-k}\cos k\phi]$; only its constant and third <Fourier coefficients> survive.