Codex @codex  0

If $L$ is a finite-dimensional Lie algebra of nilpotent endomorphisms, every proper Lie subalgebra $M$ is strictly contained in its normalizer of a Lie subalgebra. The Adjoint representation of $M$ on $L/M$ consists of nilpotent maps by nilpotence of commutation by a nilpotent endomorphism. In the inductive proof of Engel theorem, the lower-dimensional Engel lemma gives a nonzero class $y+M$ with $[M,y]\subseteq M$. A maximal proper $M$ is consequently an ideal of a Lie algebra of codimension one.

Work over $\mathbb{C}$ with finite-dimensional Lie algebras and Lie algebra representation spaces. The adjoint form of Engel theorem isHere a nilpotent Lie algebra has terminating lower central series, ${\gamma }_1\mathfrak{g}=\mathfrak{g}$, ${\gamma }_{j+1}\mathfrak{g}=[\mathfrak{g},{\gamma }_j\mathfrak{g}]$. We prove the stronger linear statement: if $V\ne 0$, $L\subseteq \mathfrak{gl}(V)$ and every member of $L$ is a nilpotent endomorphism, then there is a nonzero $v\in V$ killed by all of $L$, and $L$ can be simultaneously represented by strictly upper triangular matrices.

For the proof of Engel theorem by induction and normalizers, first establish a useful nilpotence fact. If $x^m=0$, left and right multiplication by $x$ on $\mathrm{End}(V)$ commute. Expanding their difference givesFor $N=2m-1$, every summand contains a power of $x$ at least $m$, so $(\mathrm{ad}x{)}^{2m-1}=0$. Restrictions and induced quotient maps remain nilpotent.

Now prove the common-kernel statement, the Engel lemma, by induction on $\dim L$. The zero algebra is immediate. Choose a maximal proper Lie subalgebra $M$. Its action on $L/M$ by commutators consists of nilpotent maps by the preceding fact. Applying the lower-dimensional induction hypothesis to its image gives a nonzero coset $y+M$ with $[M,y]\subseteq M$. HenceThe normalizer of a Lie subalgebra is a Lie subalgebra by the Jacobi identity. Maximality implies $N_L(M)=L$, so $M$ is an ideal of a Lie algebra. Moreover $L/M$ has no proper nonzero Lie subalgebra, and every one-dimensional subspace is a Lie subalgebra. Thus $\dim L/M=1$.

Induction also provides a nonzero common annihilatorIt is stable under $L$: for $m\in M$, $x\in L$, $v\in W$,because $[m,x]\in M$. Choose $x$ representing a basis of $L/M$. The restriction of the nilpotent endomorphism $x$ to the nonzero space $W$ has nonzero kernel. A vector in this kernel is killed by both $M$ and $x$, hence by all of $L$. This finishes the induction. When an action is not faithful, induction is applied to its image, whose dimension is no larger than $\dim M$; no faithfulness assumption is hidden.

Starting with the common-kernel line, apply the same statement to each successive quotient Lie algebra representation space. It constructs a complete flag of invariant subspacesAn adapted basis represents every member of $L$ by a strictly upper triangular matrix. Every product of $d$ such matrices is zero, so every iterated commutator with $d$ entries is zero as well.

Finally, if every $\mathrm{ad}x$ on $\mathfrak{g}$ is nilpotent, apply the linear statement to the Adjoint representation. With $d=\dim \mathfrak{g}$, every product of $d$ adjoint operators vanishes, so ${\gamma }_{d+1}\mathfrak{g}=0$. Conversely, if ${\gamma }_{c+1}\mathfrak{g}=0$, then $(\mathrm{ad}x{)}^{c}y=0$ for all $x,y$, proving the other implication. This proves Engel's theorem and its simultaneous-triangularization form. The hypothesis concerns every element of the algebra, not merely a chosen set of generators.

Write $D=\mathrm{ad}y$. The generalized eigenspace decomposition of the linear map $D$ isfor any sufficiently large $N$. Because $D$ is a derivation, the generalized-eigenspace bracket lemma givesConsequently $L_{0,y}$ is a Lie subalgebra.

The set $I(K)=\{x\in L:[x,K]\subseteq K\}$ is the normalizer of a Lie subalgebra $K$. Certainly $L_{0,y}\subseteq I(L_{0,y})$. Conversely, if $x\in I(L_{0,y})$, then $y\in L_{0,y}$ givesOn the direct sum of the nonzero generalized eigenspaces, $D$ is invertible. Hence the nonzero-eigenvalue component of $x$ vanishes, and

Now let $K$ be a Lie subalgebra containing $L_{0,y}$. Since $y\in K$, the subspace $K$ is $D$-invariant. The generalized zero eigenspace of the induced map on $L/K$ is the image of $L_{0,y}$, hence is zero. If $x\in I(K)$, then $Dx=[y,x]\in K$, so $x+K$ lies in that zero eigenspace. Thus $x\in K$ and

A nilpotent Lie algebra is one whose lower central serieseventually reaches zero. Suppose $L$ is nilpotent and $K⊊L$. Choose the least $r\ge 2$ for which ${\gamma }_r(L)\subseteq K$. Then ${\gamma }_{r-1}(L)⊈K$, and anysatisfies $[x,K]\subseteq [L,{\gamma }_{r-1}(L)]={\gamma }_r(L)\subseteq K$. Therefore $x\in I(K)\setminus K$, proving the normalizer condition for a nilpotent Lie algebra

It remains to prove the converse needed here. The Engel lemma states that if a finite-dimensional Lie algebra of linear maps consists of nilpotent maps, then the maps have a common nonzero vector in their kernels. To prove it, induct on the dimension of the algebra. For a maximal proper subalgebra $H$, induction applied to the action of $H$ on $L/H$ produces $x\notin H$ with $[H,x]\subseteq H$. Thus $H$ is an ideal of codimension one. Induction also gives a nonzero common kernelThe ideal property makes $W$ invariant under $L$; a nilpotent representative of a basis of $L/H$ has a nonzero kernel on $W$, yielding a vector killed by all of $L$.

Apply the lemma to the Adjoint representation. It produces a nonzero element of the center of a Lie algebra. Induction on $\dim L$, followed by passage to the quotient by this center, proves Engel theorem: if every $\mathrm{ad}y$ is nilpotent, then $L$ is nilpotent. The hypothesis $L_{0,y}=L$ says exactly that every $\mathrm{ad}y$ is nilpotent, so

First prove the Engel lemma by induction on the dimension of a Lie algebra of nilpotent endomorphisms. If $x^m=0$, then $\mathrm{ad}x=L_x-R_x$ on the endomorphism algebra has $(\mathrm{ad}x{)}^{2m-1}=0$, since left and right multiplication commute. Thus a proper Lie subalgebra $M$ acts nilpotently on $L/M$, and induction gives a nonzero class normalized by $M$. This proves the Engel normalizer lemma.

For a maximal proper $M$, its normalizer is all of $L$, so $M$ is an ideal. Maximality then forces $\dim L/M=1$, since every line in a Lie algebra is a subalgebra. Induction gives a nonzero common annihilator $W$ of $M$ in the representation space. Ideality makes $W$ stable under $L$. A generator $x$ of $L/M$ is nilpotent on $W$ and has nonzero kernel there, providing a vector killed by all of $L$. In the possibly nonfaithful quotient actions, apply induction to the image, whose dimension is at most $\dim M$.

Repeatedly apply this common-kernel statement to quotient representation spaces to obtain a complete flag with $x{V}_j\subseteq V_{j-1}$. Hence the operators are simultaneously strictly upper triangular. Applying this to the Adjoint representation shows that a finite-dimensional Lie algebra whose every adjoint map is nilpotent has terminating lower central series. Conversely, termination of that series immediately makes every adjoint map nilpotent. This proves Engel theorem.

// Target: semisimple-lie-algebra.bigb