 Solution

Nilpotence and eigenvalues. If $Av=\lambda v$ with $v\ne 0$, then $A^{n}v={\lambda }^{n}v$. Thus $A^n=0$ forces $\lambda =0$ for every eigenvalue. Conversely, if all $n$ complex eigenvalues, counted with algebraic multiplicity, vanish, the monic characteristic polynomial is ${\chi }_A(t)=t^n$. The Cayley-Hamilton theorem then gives $A^n=0$. HenceThis is the criterion for a nilpotent matrix.

Symmetric case. Here a symmetric matrix is understood to be real. The spectral theorem gives $A=QD{Q}^T$ with $Q$ orthogonal and $D$ real diagonal. The previous result makes every diagonal entry zero, so $A=0$. Alternatively, choose $2^r\ge n$, so $A^{2^r}=0$. For every real vector $x$,It follows that $A^{2^{r-1}}=0$. Repeat until $A=0$. This uses positivity of the quadratic form $x^T{M}^{T}Mx$. The restriction to real symmetric, or complex Hermitian matrices, matters: a complex matrix satisfying only $A^T=A$ need not have this positivity.

The parameters and powers. The first two columns of $T$ coincide, so $\det T=0$ for all $\alpha ,\beta$. The sum of its principal two-by-two determinants, each a minor of a matrix, isTogether with $\mathrm{tr}T=2+\beta$, this givesFor every eigenvalue to vanish both remaining coefficients must be zero. ThereforeDirect multiplication at these values givesIndeed the first row of $T^2$ multiplied by $T$ is $3{\mathrm{row}}_1(T)+3{\mathrm{row}}_2(T)-6{\mathrm{row}}_3(T)=0$, and the second row is its negative.

Geometry of repeated application. For any parameters the image of a linear map lies in the span of $a=(1,1,1{)}^T$ and $b=(1,\alpha ,\beta {)}^T$, since $Tx=(x_1+x_2)a+x_{3}b$. Thus it lies in a fixed plane through the origin, independently of the input. When $a,b$ are independent its normal is $a\times b=(\beta -\alpha ,1-\beta ,\alpha -1)$; if they coincide, the image is a line and is still contained in a plane.

For $\alpha =-5,\beta =-2$, put $s=x_1+x_2$ and $t=x_3$. ThenThe columns span this entire plane, so $T$ has rank two. The next two steps areThus the image of a linear map contracts from a plane to a line and then to the origin under successive applications. The line from $T^2$ is precisely the kernel of $T$.