 Solution

Preserving the Euclidean scalar product means $(Ax{)}^T(Ay)=x^{T}y$ for every pair of real vectors $x,y$. Since these bilinear expressions agree on all pairs of basis vectors, this is equivalent to $A^{T}A=I$: the columns of $A$ form an orthonormal basis.

The first column is a unit vector, so it is $(\cos \theta ,\sin \theta {)}^T$ for some real $\theta$. A perpendicular unit vector is one of the two choices $\epsilon (-\sin \theta ,\cos \theta {)}^T$, where $\epsilon \in \{1,-1\}$. Hence all the required orthogonal matrices areHere $R(\theta )$ is a planar rotation, and $\theta$ is determined modulo $2\pi$. To check the group laws explicitly, $D_{\epsilon }R(\varphi )=R(\epsilon \varphi )D_{\epsilon }$, soThis proves closure. The identity is $R(0)D_1=I$, the inverse of $R(\theta )D_{\epsilon }$ is $R(-\epsilon \theta )D_{\epsilon }$, and associativity is inherited from matrix multiplication. Thus the set is the orthogonal group $O(2)$.