 Solution

An orthogonal transformation preserves the real inner product. If $W$ is an invariant subspace, its orthogonal complement is invariant too: $\alpha W=W$ in finite dimensions, and for $v\in W^{\perp }$, $w\in W$, we have $⟨\alpha v,w⟩=⟨v,{\alpha }^{-1}w⟩=0$.

Complexify the space and the transformation. There is a complex eigenvalue $\lambda$, and preservation of the Hermitian norm gives $|\lambda |=1$. If $\lambda$ is real, it is $1$ or $-1$, and the real kernel of $\alpha -\lambda I$ contains a nonzero vector. Its span is an invariant line on which $\alpha$ acts as $+1$ or $-1$.

If $\lambda$ is nonreal, write an eigenvector as $u+iv$. Its real and imaginary parts are linearly independent: otherwise a nonzero real vector would have the nonreal eigenvalue. The real plane $W=\mathrm{span}(u,v)$ is invariant, because comparison of real and imaginary parts in $\alpha (u+iv)=\lambda (u+iv)$ gives real combinations of $u,v$. The restriction has determinant $\lambda \bar{\lambda }=1$. In an oriented orthonormal basis of $W$, it is therefore a rotationThere is no invariant real line in this plane: such a line would supply a real eigenvalue, whereas its two eigenvalues are nonreal.

Apply the argument repeatedly to the invariant orthogonal complement. Thus the smallest mutually orthogonal invariant blocks are lines with action $\pm 1$, and planes with a genuine rotation. Planes with angle $0$ or $\pi$ can be decomposed further into lines.

In dimension three a suitably chosen orthonormal basis therefore gives eitherorChanging the orientation of the plane reverses the angle, so this angle range loses no cases. These exhaust rotations, reflections and their possible compositions in three dimensions.