 Solution

Cosets and counting. A left coset of a subgroup $H$ in a group $G$ is $gH=\{gh:h\in H\}$. The left cosets partition $G$ and each has $|H|$ elements. Lagrange's theorem states that for a finite group, $\boxed{|G|=[G:H]|H|}$; in particular, the subgroup order divides the group order.

The matrix group. Work throughout in the finite field ${\mathbb{F}}_5$, so every nonzero entry has a multiplicative inverse. Denote a matrix by $A(a,b,c)$, with $a,c\ne 0$. Ordinary matrix multiplication, interpreted using modular arithmetic, givesIts diagonal entries remain nonzero, proving closure. Associativity is inherited from matrix multiplication, $A(1,0,1)$ is the identity element, andThis lies in the same set and multiplication checks the inverse in either order. Thus these matrices form a group. There are four choices each for $a$ and $c$ and five for $b$, giving $\boxed{|G|=4\cdot 5\cdot 4=80}$.

The subgroup. Write $h(a,c)=A(a,a-c,c)$. ThenIt contains the identity $h(1,1)$, is closed, and has inverse $h(a^{-1},c^{-1})$. Interchanging the two factors leaves the product unchanged because multiplication in ${\mathbb{F}}_5$ is commutative. Therefore $H$ is an abelian subgroup, withEquivalently $H$ is isomorphic to ${\mathbb{F}}_5^{\times }\times {\mathbb{F}}_5^{\times }$ by $h(a,c)\mapsto (a,c)$, which also explains its abelian property. Its order divides $80$, verifying Lagrange's theorem.

There are five distinct left cosets. Representatives can be taken as $U_t=A(1,t,1)$, $t\in {\mathbb{F}}_5$: indeedEvery matrix lies in exactly the coset specified by this unique $t$.

Elements whose square is the identity. Squaring givesThus $a,c\in \{1,-1\}=\{1,4\}$ and $b(a+c)=0$. For equal signs, $a+c\ne 0$ in ${\mathbb{F}}_5$, so $b=0$. For opposite signs, $a+c=0$ and all five $b$ values work. The complete set is thereforeIt contains $2+5+5=12$ elements, including the identity; the other eleven are involutions. Since $12$ does not divide $80$, Lagrange's theorem proves that this set is not a subgroup.