 Solution

Squaring with the product law givesEquality to the identity matrix requires $2a=2c=0$ in ${\mathbb{F}}_5$. Since 2 has a multiplicative inverse modulo 5, this forces $a=c=0$. The middle condition then gives $2b=0$, hence $b=0$. ThereforeThis agrees with Lagrange's theorem: a group of odd order has no element of order two.

The subset specified by $b\ne 0$ is not a subgroup. It omits the identity element. It also fails closure: both $A(0,1,0)$ and $A(0,4,0)$ lie in it, but their product is $A(0,0,0)=I$, whose $b$ coordinate is zero.