Past exam of the mathematics course of the University of Cambridge / 2015 / iii / Paper 2 / 1 / Solution

A nilpotent endomorphism $x$ satisfies $x^N=0$ for some positive integer $N$. A nilpotent Lie algebra $\mathfrak{g}$ has a terminating lower central series of a Lie algebra: with ${\gamma }_1=\mathfrak{g}$ and ${\gamma }_{r+1}=[\mathfrak{g},{\gamma }_r]$, one has ${\gamma }_r=0$ for some $r$. These are different conditions: the first concerns a particular linear map, whereas the second concerns repeated Lie brackets.

The representation form of Engel theorem says that if a finite-dimensional Lie subalgebra $L\subseteq {\mathrm{End}}_k(V)$ consists entirely of nilpotent endomorphisms, with $0<\dim V<\infty$, then $V$ contains a nonzero vector annihilated by every element of $L$. Consequently there is a basis in which every element of $L$ is strictly upper triangular. No assumption on the characteristic of a field or algebraic closure is required.

We prove the common-vector assertion by induction on $\dim L$, simultaneously for every nonzero finite-dimensional representation space. The zero algebra is immediate. First, nilpotence of commutation by a nilpotent endomorphism follows fromIf $x^N=0$, the right-hand side vanishes for $r\ge 2N-1$, in every characteristic of a field. For any proper Lie subalgebra $M⊊L$, its Adjoint representation on $L/M$ therefore consists of nilpotent endomorphisms. Its image has dimension at most $\dim M<\dim L$, so the inductive assertion gives a nonzero class $y+M$ annihilated by $M$. Equivalently $y\notin M$ and $[M,y]\subseteq M$. Thus the Engel normalizer lemma gives $M⊊N_L(M)$.

Choose a maximal proper Lie subalgebra $M$. Its normalizer of a Lie subalgebra must be all of $L$, so $M$ is an ideal of a Lie algebra. Moreover $L/M$ has dimension one: otherwise a one-dimensional Lie subalgebra of this quotient would have a proper inverse image strictly between $M$ and $L$. By induction the space $W=\{v\in V:Mv=0\}$ is nonzero. It is invariant under $L$, since $m(xw)=x(mw)+[m,x]w=0$. Write $L=M+kx$. The restriction of the nilpotent endomorphism $x$ to $W$ has a nonzero kernel, and any vector in that kernel is annihilated by all of $L$. This completes the induction. Apply the assertion repeatedly to $V/kv$ to obtain a complete invariant flag $0=V_0\subset V_1\subset \cdots \subset V_d=V$ with $L{V}_j\subseteq V_{j-1}$. A basis adapted to that flag makes every matrix strictly upper triangular.

A nilpotent abstract Lie algebra need not act by nilpotent matrices. For $V\ne 0$, the scalar algebra $kI\subseteq \mathfrak{gl}(V)$ is an abelian Lie algebra, hence a nilpotent Lie algebra, but $I^N=I\ne 0$. No change of basis makes $I$ strictly upper triangular. The failed implication is precisely the distinction described in nilpotent Lie algebras need not act nilpotently.

If $\mathfrak{g}$ is a nilpotent Lie algebra, then $(\mathrm{ad}x{)}^r\mathfrak{g}\subseteq {\gamma }_{r+1}$, so every $\mathrm{ad}x$ is a nilpotent endomorphism. Conversely, apply Engel theorem to $\mathrm{ad}\mathfrak{g}$ acting on $\mathfrak{g}$. Its invariant flag is lowered by every Adjoint representation matrix, so every product of $d=\dim \mathfrak{g}$ such matrices vanishes. Since ${\gamma }_{d+1}$ is spanned by expressions $\mathrm{ad}{x}_1\cdots \mathrm{ad}{x}_d(y)$, the lower central series of a Lie algebra terminates. ThereforeFinally suppose $k$ is an algebraically closed field. If some $\mathrm{ad}x$ is not a nilpotent endomorphism, it has a nonzero eigenvalue $\lambda$ and an eigenvector $y$, giving $[x,y]=\lambda y$. The vectors $x,y$ are linearly independent and span a nonabelian two-dimensional Lie subalgebra. Conversely, a Lie subalgebra of a nilpotent Lie algebra is nilpotent because its lower central series of a Lie algebra lies termwise in the ambient series. A nonabelian two-dimensional Lie algebra has a basis $u,v$ with $[u,v]=v$: choose $v$ spanning the nonzero derived algebra and rescale a complementary vector. Its Adjoint representation has the nonzero eigenvalue $1$, so it is not nilpotent. This proves the two-dimensional subalgebra criterion for Lie algebra nilpotence: