Past exam of the mathematics course of the University of Cambridge / 2009 / iii / Paper 27 / 3 / c / Solution

Set $C=AB=\left(\begin{array}{cc}0& -1\\ 1& 2\lambda \end{array}\right)$. It fixes $z=-\lambda +i\sqrt{1-{\lambda }^2}$ because $z^2+2\lambda z+1=0$. Write $\lambda =\cos \theta$, $0<\theta <\pi /2$; the eigenvalues of $C$ are $e^{\pm i\theta }$.

The stabilizer of an interior point of $\mathbb{H}$ in $PS{L}_2(\mathbb{R})$ is conjugate to the compact rotation group. Its intersection with a discrete subgroup is finite: otherwise compactness gives accumulating elements, and their quotients accumulate at the identity. Thus $C$ has finite order in the projective group. Its eigenvalues then satisfy $e^{2im\theta }=1$ for some positive integer $m$, forcing $\theta /\pi \in \mathbb{Q}$. Hence $\boxed{\lambda =\cos (\pi y)\text{ for some }y\in \mathbb{Q}}$. This is a necessary condition only.