Invariant subspace

An invariant subspace of a linear map $A:V\to V$ is a vector subspace $W\subseteq V$ satisfying $A(W)\subseteq W$. For a group representation, invariance means $\rho (g)W\subseteq W$ for every represented group element $g$.

Invariance of a vector subspace does not mean that each individual vector stays fixed. The zero-sum plane in the three-coordinate example is invariant even though the cyclic permutation rotates its nonzero vectors. By contrast, the constant-coordinate line is pointwise fixed. Distinguishing these statements prevents a symmetry-preserved mode space from being mistaken for a static mode.

For a quarter-turn rotation $T$ of ${\mathbb{R}}^3$ about a unit axis vector $n$, the only real invariant subspaces are $\{0\}$, the axis $\mathbb{R}n$, its perpendicular plane $n^{\perp }$, and all of ${\mathbb{R}}^3$.

Indeed $T^2$ fixes the axis and negates the perpendicular plane. Hence $P=(I+T^2)/2$ is the linear projection onto the axis, while $I-P$ is the linear projection onto the perpendicular plane. An invariant subspace $L$ is preserved by these polynomials in $T$, soThe first intersection is either zero or the axis. If the second contains a nonzero $v$, it also contains $Tv$. These are perpendicular nonzero vectors, so they span the perpendicular plane. This proves the classification.

Over the complex numbers the situation changes: the rotation matrix has eigenvalues $1,i,-i$, giving three one-dimensional eigenspaces. The absence of real eigenvalues on the perpendicular plane explains why it has no real one-dimensional invariant subspace.

For a linear group representation, the common fixed vectors of a subgroup form an invariant subspace for any equivariant map restricted to it. Its dimension is central to symmetry-based branching theorems. It need not be invariant under the entire group; conjugate subgroups have conjugate fixed-point subspaces.

A nonzero representation is reducible when it has a nonzero proper invariant subspace; otherwise it is an irreducible representation.