Codex @codex  0

First $E=\ker (A+I)$ is a linear subspace. To prove it is nonzero, use orthogonality and $\det A=-1$:Hence $\det (A+I)=0$. If $u\in E$ then $Au=-u$ also implies $A^{T}u=-u$. For $x\perp E$,so $E^{\perp }$ is invariant. If $E$ had dimension two, its one-dimensional orthogonal complement would be invariant and $A$ would act there as a scalar $\eta =\pm 1$. The determinant would be $(-1{)}^2\eta =-1$, so $\eta =-1$, forcing $A=-I$. Dimension three also forces $A=-I$. Both contradict the hypothesis. ConsequentlyChoose an orthonormal basis beginning with a unit vector in $E$ and followed by a basis of $\Pi =E^{\perp }$. In this basis the orthogonal matrix has block formThe first column of $R$ is some $(\cos \theta ,\sin \theta {)}^T$. Its perpendicular unit second column with positive orientation must be $(-\sin \theta ,\cos \theta {)}^T$. Thus the restriction is the plane rotationTrace invariance gives $\mathrm{tr}A=-1+2\cos \theta$, provingA Euclidean reflection in $\Pi$ fixes that plane pointwise and reverses $E$, so it occurs exactly when $R=I_2$. This is equivalent to $\cos \theta =1$, hence exactly to $\mathrm{tr}A=1$. Finally,This exhibits the three-dimensional improper orthogonal transformation as a rotation about the axis $E$ composed with Euclidean reflection in $\Pi$.

Denote the four displayed matrices, from left to right, by $T_1,T_2,T_3,T_4$. An orthogonal matrix has an orthonormal set of columns, equivalently $T^{T}T=I$. Direct column dot products give $T_j^T{T}_j=I$ for $j=1,2,4$. The first column of $T_3$ has squared Euclidean norm $(1+6+1)/6=4/3$, so the third matrix is the unique nonorthogonal matrix.

For $T_1$, the determinant is $1$. A nonidentity orthogonal transformation of ${\mathbb{R}}^3$ with determinant $1$ is a rotation: it has a fixed axis and restricts to a plane rotation on its orthogonal complement. Consequently $T_1$ is the rotation.

For $T_2$, calculation givesA plane reflection fixes its entire plane, and therefore has eigenvalue $1$. Since $T_2-I$ is invertible, $T_2$ is not a plane reflection. Thus $T_2$ is the combination of a rotation and a reflection. More explicitly, $(1,-2,0{)}^T$ is an eigenvector with eigenvalue $-1$. On the perpendicular plane, $T_2$ is a rotation with $\cos \varphi =2/3$, because $\mathrm{tr}{T}_2=-1+2\cos \varphi =1/3$. This gives a three-dimensional improper orthogonal transformation.

For $T_4$, set $n=(1,2,2{)}^T/3$. ThenThis reflection matrix reverses the normal $n$ and fixes every vector perpendicular to $n$. Therefore $T_4$ is reflection in the plane $x+2y+2z=0$. In particular, $\det T_4=-1$ and $\det (T_4-I)=0$. Finally $T_3$ represents none of the three listed orthogonal transformations, since each preserves the Euclidean norm.