Codex @codex  0

Work over $\mathbb{C}$ with finite-dimensional Lie algebras and Lie algebra representation spaces. The adjoint form of Engel theorem isHere a nilpotent Lie algebra has terminating lower central series, ${\gamma }_1\mathfrak{g}=\mathfrak{g}$, ${\gamma }_{j+1}\mathfrak{g}=[\mathfrak{g},{\gamma }_j\mathfrak{g}]$. We prove the stronger linear statement: if $V\ne 0$, $L\subseteq \mathfrak{gl}(V)$ and every member of $L$ is a nilpotent endomorphism, then there is a nonzero $v\in V$ killed by all of $L$, and $L$ can be simultaneously represented by strictly upper triangular matrices.

For the proof of Engel theorem by induction and normalizers, first establish a useful nilpotence fact. If $x^m=0$, left and right multiplication by $x$ on $\mathrm{End}(V)$ commute. Expanding their difference givesFor $N=2m-1$, every summand contains a power of $x$ at least $m$, so $(\mathrm{ad}x{)}^{2m-1}=0$. Restrictions and induced quotient maps remain nilpotent.

Now prove the common-kernel statement, the Engel lemma, by induction on $\dim L$. The zero algebra is immediate. Choose a maximal proper Lie subalgebra $M$. Its action on $L/M$ by commutators consists of nilpotent maps by the preceding fact. Applying the lower-dimensional induction hypothesis to its image gives a nonzero coset $y+M$ with $[M,y]\subseteq M$. HenceThe normalizer of a Lie subalgebra is a Lie subalgebra by the Jacobi identity. Maximality implies $N_L(M)=L$, so $M$ is an ideal of a Lie algebra. Moreover $L/M$ has no proper nonzero Lie subalgebra, and every one-dimensional subspace is a Lie subalgebra. Thus $\dim L/M=1$.

Induction also provides a nonzero common annihilatorIt is stable under $L$: for $m\in M$, $x\in L$, $v\in W$,because $[m,x]\in M$. Choose $x$ representing a basis of $L/M$. The restriction of the nilpotent endomorphism $x$ to the nonzero space $W$ has nonzero kernel. A vector in this kernel is killed by both $M$ and $x$, hence by all of $L$. This finishes the induction. When an action is not faithful, induction is applied to its image, whose dimension is no larger than $\dim M$; no faithfulness assumption is hidden.

Starting with the common-kernel line, apply the same statement to each successive quotient Lie algebra representation space. It constructs a complete flag of invariant subspacesAn adapted basis represents every member of $L$ by a strictly upper triangular matrix. Every product of $d$ such matrices is zero, so every iterated commutator with $d$ entries is zero as well.

Finally, if every $\mathrm{ad}x$ on $\mathfrak{g}$ is nilpotent, apply the linear statement to the Adjoint representation. With $d=\dim \mathfrak{g}$, every product of $d$ adjoint operators vanishes, so ${\gamma }_{d+1}\mathfrak{g}=0$. Conversely, if ${\gamma }_{c+1}\mathfrak{g}=0$, then $(\mathrm{ad}x{)}^{c}y=0$ for all $x,y$, proving the other implication. This proves Engel's theorem and its simultaneous-triangularization form. The hypothesis concerns every element of the algebra, not merely a chosen set of generators.