= Solution
\b[Cosets and counting.] A <left coset> of a <subgroup> $H$ in a <group> $G$ is $gH=\{gh:h\in H\}$. The <left cosets> partition $G$ and each has $|H|$ elements. <Lagrange's theorem> states that for a finite <group>, $\boxed{|G|=[G:H]|H|}$; in particular, the <subgroup> order divides the <group order>.
\b[The matrix group.] Work throughout in the <finite field> $\mathbb F_5$, so every nonzero entry has a multiplicative inverse. Denote a matrix by $A(a,b,c)$, with $a,c\ne0$. Ordinary <matrix multiplication>, interpreted using <modular arithmetic>, gives
$$
A(a,b,c)A(d,e,f)=A(ad,ae+bf,cf).
$$
Its diagonal entries remain nonzero, proving closure. Associativity is inherited from <matrix multiplication>, $A(1,0,1)$ is the <identity element>, and
$$
\boxed{A(a,b,c)^{-1}=A(a^{-1},-a^{-1}bc^{-1},c^{-1}).}
$$
This lies in the same set and multiplication checks the inverse in either order. Thus these matrices form a <group>. There are four choices each for $a$ and $c$ and five for $b$, giving $\boxed{|G|=4\cdot5\cdot4=80}$.
\b[The subgroup.] Write $h(a,c)=A(a,a-c,c)$. Then
$$
h(a,c)h(d,f)=A(ad,a(d-f)+(a-c)f,cf)=h(ad,cf).
$$
It contains the identity $h(1,1)$, is closed, and has inverse $h(a^{-1},c^{-1})$. Interchanging the two factors leaves the product unchanged because multiplication in $\mathbb F_5$ is commutative. Therefore $H$ is an <abelian subgroup>, with
$$
\boxed{|H|=4^2=16,\qquad [G:H]=80/16=5.}
$$
Equivalently $H$ is isomorphic to $\mathbb F_5^\times\times\mathbb F_5^\times$ by $h(a,c)\mapsto(a,c)$, which also explains its abelian property. Its order divides $80$, verifying <Lagrange's theorem>.
There are five distinct <left cosets>. Representatives can be taken as $U_t=A(1,t,1)$, $t\in\mathbb F_5$: indeed
$$
U_th(a,c)=A(a,a-c+tc,c),\qquad t=(b-a+c)c^{-1}.
$$
Every matrix lies in exactly the coset specified by this unique $t$.
\b[Elements whose square is the identity.] Squaring gives
$$
A(a,b,c)^2=A(a^2,b(a+c),c^2).
$$
Thus $a,c\in\{1,-1\}=\{1,4\}$ and $b(a+c)=0$. For equal signs, $a+c\ne0$ in $\mathbb F_5$, so $b=0$. For opposite signs, $a+c=0$ and all five $b$ values work. The complete set is therefore
$$
\boxed{\{I,-I\}\ \cup\ \left\{\begin{pmatrix}1&b\\0&-1\end{pmatrix}:b\in\mathbb F_5\right\}\ \cup\ \left\{\begin{pmatrix}-1&b\\0&1\end{pmatrix}:b\in\mathbb F_5\right\}.}
$$
It contains $2+5+5=12$ elements, including the identity; the other eleven are <involutions>. Since $12$ does not divide $80$, <Lagrange's theorem> proves that this set is not a <subgroup>.
Back to article page
