Engel's theorem

A finite-dimensional Lie algebra is nilpotent if and only if every adjoint map ${\mathrm{ad}}_x$ is nilpotent. A Lie algebra of nilpotent endomorphisms can be simultaneously represented by strictly upper triangular matrices.

First prove the Engel lemma by induction on the dimension of a Lie algebra of nilpotent endomorphisms. If $x^m=0$, then $\mathrm{ad}x=L_x-R_x$ on the endomorphism algebra has $(\mathrm{ad}x{)}^{2m-1}=0$, since left and right multiplication commute. Thus a proper Lie subalgebra $M$ acts nilpotently on $L/M$, and induction gives a nonzero class normalized by $M$. This proves the Engel normalizer lemma.

For a maximal proper $M$, its normalizer is all of $L$, so $M$ is an ideal. Maximality then forces $\dim L/M=1$, since every line in a Lie algebra is a subalgebra. Induction gives a nonzero common annihilator $W$ of $M$ in the representation space. Ideality makes $W$ stable under $L$. A generator $x$ of $L/M$ is nilpotent on $W$ and has nonzero kernel there, providing a vector killed by all of $L$. In the possibly nonfaithful quotient actions, apply induction to the image, whose dimension is at most $\dim M$.

Repeatedly apply this common-kernel statement to quotient representation spaces to obtain a complete flag with $x{V}_j\subseteq V_{j-1}$. Hence the operators are simultaneously strictly upper triangular. Applying this to the Adjoint representation shows that a finite-dimensional Lie algebra whose every adjoint map is nilpotent has terminating lower central series. Conversely, termination of that series immediately makes every adjoint map nilpotent. This proves Engel theorem.

// Target: semisimple-lie-algebra.bigb

A finite-dimensional Lie algebra representation in which every representing endomorphism is nilpotent has a nonzero vector annihilated by the entire Lie algebra. Applying this to the Adjoint representation is the key step in Engel theorem.

If $L$ is a finite-dimensional Lie algebra of nilpotent endomorphisms, every proper Lie subalgebra $M$ is strictly contained in its normalizer of a Lie subalgebra. The Adjoint representation of $M$ on $L/M$ consists of nilpotent maps by nilpotence of commutation by a nilpotent endomorphism. In the inductive proof of Engel theorem, the lower-dimensional Engel lemma gives a nonzero class $y+M$ with $[M,y]\subseteq M$. A maximal proper $M$ is consequently an ideal of a Lie algebra of codimension one.