Solution (source code)

= Solution

The <spectral theorem> gives a real orthonormal <eigenbasis> $v_1,\ldots,v_n$ for a real <symmetric matrix> $A$. Distinct <eigenspaces> are <orthogonal>; within a repeated <eigenspace>, choose an <orthonormal basis>, for example by <Gram-Schmidt orthogonalization>. Put these <eigenvectors> in the columns of $T$. Then $T^{-1}=T^T$ and
$$
T^{-1}AT=\operatorname{diag}(\lambda_1,\ldots,\lambda_n).
$$
For a change of variables $y=Sx$, choose $\boxed{S=T^T}$. Since $x=Ty$, the <quadratic form> becomes $x^TAx=\sum_i\lambda_i y_i^2$; the $\lambda_i$ are precisely the <eigenvalues> of $A$.

In the numerical case, the off-diagonal entries are half the cross-term coefficients, so
$$
A=\begin{pmatrix}1&1&3\\1&1&-3\\3&-3&-3\end{pmatrix}.
$$
A direct <eigenvector> calculation gives the unit <eigenvectors> $(1,1,0)/\sqrt2$, $(1,-1,1)/\sqrt3$, $(1,-1,-2)/\sqrt6$, with <eigenvalues> two, three and minus six. They are pairwise <orthogonal>. Therefore the required <orthogonal diagonalization> is
$$
\boxed{y_1=\frac{x_1+x_2}{\sqrt2},\qquad y_2=\frac{x_1-x_2+x_3}{\sqrt3},\qquad y_3=\frac{x_1-x_2-2x_3}{\sqrt6},}
$$
and the transformed <quadratic form> is $\boxed{2y_1^2+3y_2^2-6y_3^2}$.