Past exam of the mathematics course of the University of Cambridge / 2005 / ii / Paper 1 / 12G / Solution

In the hyperbolic-ball model, the limit set of a Kleinian group is the boundary accumulation set of an orbit $Go$ of an interior point: $\Lambda (G)=\overline{Go}\cap \partial {\mathbb{H}}^3$. The choice of $o$ does not matter, since any two such orbits remain a fixed hyperbolic distance apart. It is closed and $G$-invariant. The complicated/non-elementary hypothesis excludes relatively compact groups and elementary groups with at most two limit points; in particular the limit set is nonempty. Having no finite boundary orbit alone would not exclude a compact rotation group, whose interior orbits are bounded.

If $\Lambda (G)$ were finite and nonempty, the orbit of any of its points would be a finite boundary orbit, contrary to the hypothesis. Thus it is infinite. In fact it cannot be countable. We give the essential minimality/perfectness argument. A sequence of Möbius transformations sending an interior point toward $\xi \in \Lambda$ has a subsequence converging on the boundary away from one exceptional point to the constant $\xi$. This follows by normalizing the representing $2\times 2$ matrices: a diverging sequence has a rank-one projective limit, with one kernel line and one image line. For any boundary point $\eta$, the absence of a finite orbit allows choosing $h\in G$ with $h\eta$ different from that exceptional point; then $g_{j}h\eta \to \xi$. Hence the orbit of every $\eta \in \Lambda$ is dense in $\Lambda$.

If $\xi$ were an isolated limit point, density of every orbit would make every point belong to its orbit: each orbit must hit the open singleton $\{\xi \}$. Every point would then be isolated by equivariance, making the compact set $\Lambda$ discrete and therefore finite, a contradiction. Thus $\Lambda$ is a nonempty perfect compact metric space. Repeatedly choosing two disjoint smaller neighbourhoods meeting this perfect set, with diameters tending to zero, associates distinct nested limits to all infinite binary sequences. The limit set is therefore uncountable.

Jorgensen inequality states that for a non-elementary discrete group generated by lifts $A,B\in \mathrm{SL}(2,\mathbb{C})$,For a concrete failure of discreteness, take $A=\mathrm{diag}(e^{s/2},e^{-s/2})$ and $B=\left(\begin{array}{cc}1& 1\\ -1& 0\end{array}\right)$ with $s>0$ small. The fixed points of $A$ and of $BA{B}^{-1}$ collectively include $0,\infty ,-1$, so the group is non-elementary. Direct multiplication gives both summands equal to $4{\sinh }^2(s/2)$. Their sum is less than one when $s$ is sufficiently small, proving this two-generator group is not discrete.

For a connected-limit-set example, the modular group generated by $z\mapsto z+1$ and $z\mapsto -1/z$ is discrete because its matrices have integer entries. It preserves the upper half-plane; its limit set is the extended real line, a circle. Indeed the orbit of the parabolic fixed point infinity consists of the rational points and infinity, dense in that circle. For a disconnected example, choose four pairwise disjoint round disks and two loxodromic maps pairing the exterior of each negative disk with the interior of its positive disk. The Schottky group generated by these maps is free and discrete by the disjoint-domain ping-pong construction. Successive reduced words give nested disjoint disks with shrinking diameters; their limiting points form a Cantor set. Thus its limit set is disconnected, unlike that of the modular group.