 Solution

Write a homogeneous quadratic polynomial as $p(x,y)=a{x}^2+bxy+c{y}^2$. Invariance under $B:(x,y)\mapsto (x,-y)$ forces $b=0$. Invariance under $A:(x,y)\mapsto (y,x)$ then forces $a=c$. Since $A,B$ generate the group, this proves that all homogeneous quadratic polynomial invariants areThis agrees with the single trivial representation in the symmetric square in solution; more precisely, coordinate polynomials lie in the symmetric square of the dual representation, which here is equivalent to $\rho$. If “quadratic” permits terms of smaller degree, arbitrary constants can be added. There are no invariant linear terms because $-I\in G$ negates every linear polynomial.