Solution (source code)

= Solution

Denote the four displayed matrices, from left to right, by $T_1,T_2,T_3,T_4$. An <orthogonal matrix> has an orthonormal set of columns, equivalently $T^TT=I$. Direct column <dot products> give $T_j^TT_j=I$ for $j=1,2,4$. The first column of $T_3$ has squared <Euclidean norm> $(1+6+1)/6=4/3$, so \b[the third matrix is the unique nonorthogonal matrix].

For $T_1$, the <determinant> is $1$. A nonidentity <orthogonal transformation> of $\mathbb R^3$ with <determinant> $1$ is a rotation: it has a fixed axis and restricts to a plane rotation on its <orthogonal complement>. Consequently \b[$T_1$ is the rotation].

For $T_2$, calculation gives
$$
\det T_2=-1,\qquad \det(T_2-I)=-\frac43.
$$
A plane <reflection (mathematics)> fixes its entire plane, and therefore has <eigenvalue> $1$. Since $T_2-I$ is invertible, $T_2$ is not a plane <reflection (mathematics)>. Thus \b[$T_2$ is the combination of a rotation and a reflection]. More explicitly, $(1,-2,0)^T$ is an <eigenvector> with <eigenvalue> $-1$. On the perpendicular plane, $T_2$ is a rotation with $\cos\phi=2/3$, because $\operatorname{tr}T_2=-1+2\cos\phi=1/3$. This gives a <three-dimensional improper orthogonal transformation>.

For $T_4$, set $n=(1,2,2)^T/3$. Then
$$
T_4=I-2nn^T.
$$
This <reflection matrix> reverses the normal $n$ and fixes every vector perpendicular to $n$. Therefore \b[$T_4$ is reflection in the plane $x+2y+2z=0$]. In particular, $\det T_4=-1$ and $\det(T_4-I)=0$. Finally \b[$T_3$ represents none of the three listed orthogonal transformations], since each preserves the <Euclidean norm>.