Past exam of the mathematics course of the University of Cambridge / 2017 / ii / Paper 1 / 18G / b / iii

On the normal subgroup $N=⟨a,b⟩$, the first diagonal entry is the linear character $\chi (a^i{b}^j)={\epsilon }^i{\eta }^j$. Its image has order at most three, although $|N|=9$, so its kernel of a group homomorphism has order at least three. The second entry is the inverse character and has the same kernel of a group homomorphism. ThusElements of the other coset have off-diagonal image and cannot be the identity. If $(\epsilon ,\eta )\ne (1,1)$, the kernel of a group homomorphism has order exactly three; otherwise it is all of $N$.

When both roots equal one, only $C$ acts nontrivially and its eigenlines spanned by $(1,1)$ and $(1,-1)$ are invariant, so the representation is reducible. Otherwise one of $A,B$ has two distinct eigenvalues. Any invariant line would be one of its coordinate eigenlines, but $C$ interchanges these two lines. Hence no invariant line exists, and