 Solution

A continuous homomorphism $G\to O(n)$ is immediately an $n$-dimensional real group representation by composing with the inclusion $O(n)\subset G{L}_n(\mathbb{R})$.

Conversely, let $\pi :G\to GL(V)$ be a continuous real representation with ${\dim }_{\mathbb{R}}V=n$. Choose any positive-definite inner product $(,{)}_0$ on $V$ and normalized Haar measure $dg$ on $G$. DefineThis is bilinear and symmetric. It is positive definite because the continuous nonnegative integrand $(\pi (g)v,\pi (g)v{)}_0$ is positive at every $g$ whenever $v\ne 0$. Right invariance of normalized Haar measure on a compact group givesChoose an orthonormal basis for this averaged form. In that basis every $\pi (h)$ lies in $O(n)$, proving the orthogonalization of a compact-group representation and the stated equivalence.