Solution (source code)

= Solution
{c}

\b[$A$ is not an <orthogonal matrix>.] Its first column has squared <Euclidean norm>
$$
(-\sqrt2)^2+0^2+(3/\sqrt2)^2=\frac{13}{2}\ne1,
$$
so $A^TA\ne I$. The reason this is compatible with <4/c/ii> is that $1,x,x^2$ is not an <orthonormal basis> for the given <inner product>. Its <Gram matrix> is
$$
G=\begin{pmatrix}1&0&2/3\\0&2/3&0\\2/3&0&2/3\end{pmatrix},\qquad
\langle u,v\rangle=[u]^TG[v].
$$
The correct condition on this <matrix representation of a linear map> is
$$
\boxed{A^TGA=G,}
$$
which follows either by direct multiplication or by the <inner product> preservation just proved. This is the rule for an <orthogonal transformation in a nonorthonormal basis>.