Solution (source code)

= Solution

A continuous homomorphism $G\to O(n)$ is immediately an $n$-dimensional real <group representation> by composing with the inclusion $O(n)\subset GL_n(\mathbb R)$.

Conversely, let $\pi:G\to GL(V)$ be a continuous real representation with $\dim_{\mathbb R}V=n$. Choose any positive-definite inner product $(\ ,\ )_0$ on $V$ and normalized <Haar measure> $dg$ on $G$. Define
$$
(v,w)_G=\int_G(\pi(g)v,\pi(g)w)_0\,dg.
$$
This is bilinear and symmetric. It is positive definite because the continuous nonnegative integrand $(\pi(g)v,\pi(g)v)_0$ is positive at every $g$ whenever $v\ne0$. Right invariance of normalized Haar measure on a compact group gives
$$
(\pi(h)v,\pi(h)w)_G
=\int_G(\pi(gh)v,\pi(gh)w)_0\,dg
=(v,w)_G.
$$
Choose an orthonormal basis for this averaged form. In that basis every $\pi(h)$ lies in $O(n)$, proving the <orthogonalization of a compact-group representation> and the stated equivalence.