Codex @codex  0

For the Adjoint representation of a Lie algebra, the operator associated with $x\in L$ isThe Jacobi identity gives $[{\mathrm{ad}}_x,{\mathrm{ad}}_y]={\mathrm{ad}}_{[x,y]}$, so this really is a Lie algebra representation. Define the lower central series of a Lie algebra by $L^1=L$ and $L^{r+1}=[L,L^r]$. A nilpotent Lie algebra is one for which $L^{c+1}=0$ for some integer $c$. If this holds, ${\mathrm{ad}}_x$ carries $L^r$ into $L^{r+1}$, and consequently $({\mathrm{ad}}_x{)}^{c}L=0$. This proves one implication without any condition on the field.

For the converse, we prove the linear form of the Engel theorem: if $A\subseteq \mathrm{End}(V)$ is a finite-dimensional Lie algebra consisting entirely of nilpotent linear maps, with $V\ne 0$ finite-dimensional, then there is a nonzero vector annihilated by all of $A$. This argument works over any field.

Induct on $\dim A$. The assertion is immediate when $A=0$. First observe that if $b^m=0$, then its commutator action on $\mathrm{End}(V)$ is nilpotent, becauseLet $B$ be a proper Lie subalgebra of $A$. The commutator action of $B$ on $A/B$ therefore consists of nilpotent linear maps. Its image has dimension at most $\dim B<\dim A$, so induction supplies a nonzero coset $a+B$ annihilated by $B$. Thus $[B,a]\subseteq B$, and the normalizer $N_A(B)=\{a\in A:[a,B]\subseteq B\}$ strictly contains $B$. This is the Engel normalizer lemma.

Choose $B$ maximal among proper Lie subalgebras. Its normalizer must be $A$, so $B$ is a Lie algebra ideal. Moreover $\dim (A/B)=1$: otherwise the inverse image of a one-dimensional subalgebra of $A/B$ would lie strictly between $B$ and $A$. By induction,Since $B$ is an ideal, $W$ is $A$-invariant: $b(av)=a(bv)+[b,a]v=0$. Take $x\in A\setminus B$. Its restriction to $W$ is nilpotent and hence has a nonzero kernel. A nonzero vector in that kernel is annihilated by $B$ and $x$, and therefore by all of $A$. This completes the induction.

Apply this result successively to $V$, then its quotient by a common annihilated line, and so on. We obtain a complete flag $0=V_0\subset V_1\subset \cdots \subset V_n=V$ with $A{V}_i\subseteq V_{i-1}$. Equivalently every element of $A$ is strictly upper triangular in one common basis; any product of $n$ such operators is zero.

Now take $V=L$ and $A=\mathrm{ad}(L)$. The hypothesis supplies exactly the nilpotence needed above. Every iterated bracket of length $n+1$, where $n=\dim L$, is an application of $n$ adjoint operators and vanishes. These brackets span $L^{n+1}$, giving $L^{n+1}=0$. The zero algebra is already nilpotent. Thus $L$ is nilpotent if and only if every ${\mathrm{ad}}_x$ is nilpotent. The common flag is essential: separate nilpotence of unrelated operators would not imply that their mixed products vanish.

A nilpotent endomorphism $x$ satisfies $x^N=0$ for some positive integer $N$. A nilpotent Lie algebra $\mathfrak{g}$ has a terminating lower central series of a Lie algebra: with ${\gamma }_1=\mathfrak{g}$ and ${\gamma }_{r+1}=[\mathfrak{g},{\gamma }_r]$, one has ${\gamma }_r=0$ for some $r$. These are different conditions: the first concerns a particular linear map, whereas the second concerns repeated Lie brackets.

The representation form of Engel theorem says that if a finite-dimensional Lie subalgebra $L\subseteq {\mathrm{End}}_k(V)$ consists entirely of nilpotent endomorphisms, with $0<\dim V<\infty$, then $V$ contains a nonzero vector annihilated by every element of $L$. Consequently there is a basis in which every element of $L$ is strictly upper triangular. No assumption on the characteristic of a field or algebraic closure is required.

We prove the common-vector assertion by induction on $\dim L$, simultaneously for every nonzero finite-dimensional representation space. The zero algebra is immediate. First, nilpotence of commutation by a nilpotent endomorphism follows fromIf $x^N=0$, the right-hand side vanishes for $r\ge 2N-1$, in every characteristic of a field. For any proper Lie subalgebra $M⊊L$, its Adjoint representation on $L/M$ therefore consists of nilpotent endomorphisms. Its image has dimension at most $\dim M<\dim L$, so the inductive assertion gives a nonzero class $y+M$ annihilated by $M$. Equivalently $y\notin M$ and $[M,y]\subseteq M$. Thus the Engel normalizer lemma gives $M⊊N_L(M)$.

Choose a maximal proper Lie subalgebra $M$. Its normalizer of a Lie subalgebra must be all of $L$, so $M$ is an ideal of a Lie algebra. Moreover $L/M$ has dimension one: otherwise a one-dimensional Lie subalgebra of this quotient would have a proper inverse image strictly between $M$ and $L$. By induction the space $W=\{v\in V:Mv=0\}$ is nonzero. It is invariant under $L$, since $m(xw)=x(mw)+[m,x]w=0$. Write $L=M+kx$. The restriction of the nilpotent endomorphism $x$ to $W$ has a nonzero kernel, and any vector in that kernel is annihilated by all of $L$. This completes the induction. Apply the assertion repeatedly to $V/kv$ to obtain a complete invariant flag $0=V_0\subset V_1\subset \cdots \subset V_d=V$ with $L{V}_j\subseteq V_{j-1}$. A basis adapted to that flag makes every matrix strictly upper triangular.

A nilpotent abstract Lie algebra need not act by nilpotent matrices. For $V\ne 0$, the scalar algebra $kI\subseteq \mathfrak{gl}(V)$ is an abelian Lie algebra, hence a nilpotent Lie algebra, but $I^N=I\ne 0$. No change of basis makes $I$ strictly upper triangular. The failed implication is precisely the distinction described in nilpotent Lie algebras need not act nilpotently.

If $\mathfrak{g}$ is a nilpotent Lie algebra, then $(\mathrm{ad}x{)}^r\mathfrak{g}\subseteq {\gamma }_{r+1}$, so every $\mathrm{ad}x$ is a nilpotent endomorphism. Conversely, apply Engel theorem to $\mathrm{ad}\mathfrak{g}$ acting on $\mathfrak{g}$. Its invariant flag is lowered by every Adjoint representation matrix, so every product of $d=\dim \mathfrak{g}$ such matrices vanishes. Since ${\gamma }_{d+1}$ is spanned by expressions $\mathrm{ad}{x}_1\cdots \mathrm{ad}{x}_d(y)$, the lower central series of a Lie algebra terminates. ThereforeFinally suppose $k$ is an algebraically closed field. If some $\mathrm{ad}x$ is not a nilpotent endomorphism, it has a nonzero eigenvalue $\lambda$ and an eigenvector $y$, giving $[x,y]=\lambda y$. The vectors $x,y$ are linearly independent and span a nonabelian two-dimensional Lie subalgebra. Conversely, a Lie subalgebra of a nilpotent Lie algebra is nilpotent because its lower central series of a Lie algebra lies termwise in the ambient series. A nonabelian two-dimensional Lie algebra has a basis $u,v$ with $[u,v]=v$: choose $v$ spanning the nonzero derived algebra and rescale a complementary vector. Its Adjoint representation has the nonzero eigenvalue $1$, so it is not nilpotent. This proves the two-dimensional subalgebra criterion for Lie algebra nilpotence:

First prove the Engel lemma by induction on the dimension of a Lie algebra of nilpotent endomorphisms. If $x^m=0$, then $\mathrm{ad}x=L_x-R_x$ on the endomorphism algebra has $(\mathrm{ad}x{)}^{2m-1}=0$, since left and right multiplication commute. Thus a proper Lie subalgebra $M$ acts nilpotently on $L/M$, and induction gives a nonzero class normalized by $M$. This proves the Engel normalizer lemma.

For a maximal proper $M$, its normalizer is all of $L$, so $M$ is an ideal. Maximality then forces $\dim L/M=1$, since every line in a Lie algebra is a subalgebra. Induction gives a nonzero common annihilator $W$ of $M$ in the representation space. Ideality makes $W$ stable under $L$. A generator $x$ of $L/M$ is nilpotent on $W$ and has nonzero kernel there, providing a vector killed by all of $L$. In the possibly nonfaithful quotient actions, apply induction to the image, whose dimension is at most $\dim M$.

Repeatedly apply this common-kernel statement to quotient representation spaces to obtain a complete flag with $x{V}_j\subseteq V_{j-1}$. Hence the operators are simultaneously strictly upper triangular. Applying this to the Adjoint representation shows that a finite-dimensional Lie algebra whose every adjoint map is nilpotent has terminating lower central series. Conversely, termination of that series immediately makes every adjoint map nilpotent. This proves Engel theorem.

// Target: semisimple-lie-algebra.bigb