= Solution
A <nilpotent endomorphism> $x$ satisfies $x^N=0$ for some <positive integer> $N$. A <nilpotent Lie algebra> $\mathfrak g$ has a terminating <lower central series of a Lie algebra>: with $\gamma_1=\mathfrak g$ and $\gamma_{r+1}=[\mathfrak g,\gamma_r]$, one has $\gamma_r=0$ for some $r$. These are different conditions: the first concerns a particular <linear map>, whereas the second concerns repeated <Lie brackets>.
The representation form of <Engel theorem> says that if a finite-dimensional <Lie subalgebra> $L\subseteq\operatorname{End}_k(V)$ consists entirely of <nilpotent endomorphisms>, with $0<\dim V<\infty$, then $V$ contains a nonzero vector annihilated by every element of $L$. Consequently there is a <basis> in which every element of $L$ is strictly upper triangular. No assumption on the <characteristic of a field> or algebraic closure is required.
We prove the common-vector assertion by induction on $\dim L$, simultaneously for every nonzero finite-dimensional representation space. The zero algebra is immediate. First, <nilpotence of commutation by a nilpotent endomorphism> follows from
$$
(\operatorname{ad}x)^r(T)=\sum_{j=0}^r(-1)^j\binom rj x^{r-j}Tx^j.
$$
If $x^N=0$, the right-hand side vanishes for $r\geq2N-1$, in every <characteristic of a field>. For any proper <Lie subalgebra> $M\subsetneq L$, its <Adjoint representation> on $L/M$ therefore consists of <nilpotent endomorphisms>. Its image has dimension at most $\dim M<\dim L$, so the inductive assertion gives a nonzero class $y+M$ annihilated by $M$. Equivalently $y\notin M$ and $[M,y]\subseteq M$. Thus the <Engel normalizer lemma> gives $M\subsetneq N_L(M)$.
Choose a maximal proper <Lie subalgebra> $M$. Its <normalizer of a Lie subalgebra> must be all of $L$, so $M$ is an <ideal of a Lie algebra>. Moreover $L/M$ has dimension one: otherwise a one-dimensional <Lie subalgebra> of this quotient would have a proper inverse image strictly between $M$ and $L$. By induction the space $W=\{v\in V:Mv=0\}$ is nonzero. It is invariant under $L$, since $m(xw)=x(mw)+[m,x]w=0$. Write $L=M+kx$. The restriction of the <nilpotent endomorphism> $x$ to $W$ has a nonzero <kernel>, and any vector in that kernel is annihilated by all of $L$. This completes the induction. Apply the assertion repeatedly to $V/kv$ to obtain a complete invariant flag $0=V_0\subset V_1\subset\cdots\subset V_d=V$ with $LV_j\subseteq V_{j-1}$. A <basis> adapted to that flag makes every matrix strictly upper triangular.
\b[A nilpotent abstract <Lie algebra> need not act by nilpotent matrices.] For $V\ne0$, the scalar algebra $kI\subseteq\mathfrak{gl}(V)$ is an <abelian Lie algebra>, hence a <nilpotent Lie algebra>, but $I^N=I\ne0$. No change of <basis> makes $I$ strictly upper triangular. The failed implication is precisely the distinction described in <nilpotent Lie algebras need not act nilpotently>.
If $\mathfrak g$ is a <nilpotent Lie algebra>, then $(\operatorname{ad}x)^r\mathfrak g\subseteq\gamma_{r+1}$, so every $\operatorname{ad}x$ is a <nilpotent endomorphism>. Conversely, apply <Engel theorem> to $\operatorname{ad}\mathfrak g$ acting on $\mathfrak g$. Its invariant flag is lowered by every <Adjoint representation> matrix, so every product of $d=\dim\mathfrak g$ such matrices vanishes. Since $\gamma_{d+1}$ is spanned by expressions $\operatorname{ad}x_1\cdots\operatorname{ad}x_d(y)$, the <lower central series of a Lie algebra> terminates. Therefore
$$
\boxed{\mathfrak g\text{ is nilpotent}\quad\Longleftrightarrow\quad\operatorname{ad}x\text{ is nilpotent for every }x\in\mathfrak g.}
$$
Finally suppose $k$ is an <algebraically closed field>. If some $\operatorname{ad}x$ is not a <nilpotent endomorphism>, it has a nonzero <eigenvalue> $\lambda$ and an <eigenvector> $y$, giving $[x,y]=\lambda y$. The vectors $x,y$ are <linearly independent> and span a nonabelian two-dimensional <Lie subalgebra>. Conversely, a <Lie subalgebra> of a <nilpotent Lie algebra> is nilpotent because its <lower central series of a Lie algebra> lies termwise in the ambient series. A nonabelian two-dimensional <Lie algebra> has a <basis> $u,v$ with $[u,v]=v$: choose $v$ spanning the nonzero derived algebra and rescale a complementary vector. Its <Adjoint representation> has the nonzero <eigenvalue> $1$, so it is not nilpotent. This proves the <two-dimensional subalgebra criterion for Lie algebra nilpotence>:
$$
\boxed{\mathfrak g\text{ is nilpotent}\quad\Longleftrightarrow\quad\text{every two-dimensional subalgebra is abelian}.}
$$
Back to article page
