Solution (source code)

= Solution

A real two-dimensional <group representation> is a <group homomorphism> $D:G\to GL(2,\mathbb R)$, so $D(AB)=D(A)D(B)$ and $D(I)=I$.

Substitute $\tilde x=\alpha x+\beta y$ and $\tilde y=\gamma x+\delta y$ into the quadratic expression. Its coefficients of $x^2$ and $y^2$ are $\alpha^2\hat a+\gamma^2\hat b$ and $\beta^2\hat a+\delta^2\hat b$. The cross term is zero for every matrix in this particular <cyclic group>, because each has one nonzero entry per row. Thus
$$
\boxed{M(A)=\begin{pmatrix}\alpha^2&\gamma^2\\\beta^2&\delta^2\end{pmatrix}.}
$$
Let $J=\begin{pmatrix}0&-1\\1&0\end{pmatrix}$ generate the group. For $A=\pm I$, $M=I$; for $A=\pm J$, $M=S:=\begin{pmatrix}0&1\\1&0\end{pmatrix}$. Since $S^2=I$, each $M$ is its own inverse and $D(J^k)=S^k$. The group law is respected: $S^{k+l}=S^kS^l$, and $S^4=I$ makes it independent of the representative modulo four. This proves that $D=M^{-1}$ is a <group representation>.

This <quarter-turn action on diagonal quadratic forms> has
$$
\boxed{\ker D=\{I,-I\},}
$$
so it is \b[not faithful]. Quadratic forms cannot distinguish a simultaneous sign reversal of both coordinates. The invariant coefficient directions $(1,1)$ and $(1,-1)$ correspond to the fixed form $x^2+y^2$ and the form $x^2-y^2$ whose sign changes under a quarter-turn.