= Solution
Work over $\mathbb C$ with finite-dimensional <Lie algebras> and <Lie algebra representation> spaces. The adjoint form of <Engel theorem> is
$$
\boxed{\mathfrak g\text{ is nilpotent }\Longleftrightarrow
\operatorname{ad}x\text{ is nilpotent for every }x\in\mathfrak g.}
$$
Here a <nilpotent Lie algebra> has terminating <lower central series>, $\gamma_1\mathfrak g=\mathfrak g$, $\gamma_{j+1}\mathfrak g=[\mathfrak g,\gamma_j\mathfrak g]$. We prove the stronger linear statement: if $V\ne0$, $L\subseteq\mathfrak{gl}(V)$ and every member of $L$ is a <nilpotent endomorphism>, then there is a nonzero $v\in V$ killed by all of $L$, and $L$ can be simultaneously represented by <strictly upper triangular matrices>.
For the <proof of Engel theorem by induction and normalizers>, first establish a useful nilpotence fact. If $x^m=0$, left and right multiplication by $x$ on $\operatorname{End}(V)$ commute. Expanding their difference gives
$$
(\operatorname{ad}x)^N(T)=\sum_{j=0}^N(-1)^j\binom Njx^{N-j}Tx^j.
$$
For $N=2m-1$, every summand contains a power of $x$ at least $m$, so $(\operatorname{ad}x)^{2m-1}=0$. Restrictions and induced quotient maps remain nilpotent.
Now prove the common-kernel statement, the <Engel lemma>, by induction on $\dim L$. The zero algebra is immediate. Choose a maximal proper <Lie subalgebra> $M$. Its action on $L/M$ by commutators consists of nilpotent maps by the preceding fact. Applying the lower-dimensional induction hypothesis to its image gives a nonzero coset $y+M$ with $[M,y]\subseteq M$. Hence
$$
M\subsetneq N_L(M),\qquad
N_L(M)=\{z\in L:[z,M]\subseteq M\}.
$$
The <normalizer of a Lie subalgebra> is a <Lie subalgebra> by the <Jacobi identity>. Maximality implies $N_L(M)=L$, so $M$ is an <ideal of a Lie algebra>. Moreover $L/M$ has no proper nonzero <Lie subalgebra>, and every one-dimensional subspace is a <Lie subalgebra>. Thus $\dim L/M=1$.
Induction also provides a nonzero common annihilator
$$
W=\{v\in V:mv=0\text{ for all }m\in M\}.
$$
It is stable under $L$: for $m\in M$, $x\in L$, $v\in W$,
$$
m(xv)=x(mv)+[m,x]v=0,
$$
because $[m,x]\in M$. Choose $x$ representing a <basis> of $L/M$. The restriction of the <nilpotent endomorphism> $x$ to the nonzero space $W$ has nonzero <kernel>. A vector in this <kernel> is killed by both $M$ and $x$, hence by all of $L$. This finishes the induction. When an action is not faithful, induction is applied to its image, whose <dimension> is no larger than $\dim M$; no faithfulness assumption is hidden.
Starting with the common-kernel line, apply the same statement to each successive quotient <Lie algebra representation> space. It constructs a <complete flag> of <invariant subspaces>
$$
0=V_0\subset V_1\subset\cdots\subset V_d=V,\qquad
\dim V_j=j,\qquad LV_j\subseteq V_{j-1}.
$$
An adapted <basis> represents every member of $L$ by a <strictly upper triangular matrix>. Every product of $d$ such matrices is zero, so every iterated commutator with $d$ entries is zero as well.
Finally, if every $\operatorname{ad}x$ on $\mathfrak g$ is nilpotent, apply the linear statement to the <Adjoint representation>. With $d=\dim\mathfrak g$, every product of $d$ adjoint operators vanishes, so $\gamma_{d+1}\mathfrak g=0$. Conversely, if $\gamma_{c+1}\mathfrak g=0$, then $(\operatorname{ad}x)^cy=0$ for all $x,y$, proving the other implication. \b[This proves Engel's theorem and its simultaneous-triangularization form.] The hypothesis concerns every element of the algebra, not merely a chosen set of generators.
Back to article page
