Solution (source code)

= Solution

Write $P=\omega\omega^T$. Since $P^T=P$ and $P^2=P$, the <orthogonal reflection> $R_\omega=I-2P$ satisfies $R_\omega^TR_\omega=I$ and $R_\omega^{-1}=R_\omega$. Thus, for every <Borel set> $E$, invariance under the <orthogonal group> gives
$$
(R_\omega{}_\#\mu)(E)=\mu(R_\omega^{-1}E)=\mu(E),\qquad\boxed{R_\omega{}_\#\mu=\mu.}
$$

\b[The spherical average must use normalized surface measure without an additional area factor.] Let $d\sigma$ be the normalized <surface measure on a sphere>, and set
$$
A\eta(r)=\int_{\mathbb S^{d-1}}\eta(r\theta)\,d\sigma(\theta).
$$
The correct <radial disintegration of an orthogonally invariant measure> is
$$
\boxed{\int\eta(x)\,d\mu(x)=\int_{[0,\infty)}A\eta(r)\,d(\Phi_\#\mu)(r).}
$$
Indeed, take normalized <Haar measure> $H$ on the compact <orthogonal group> $O(d)$. Invariance and <Fubini's theorem> give
$$
\int\eta\,d\mu=\int\!\int_{O(d)}\eta(Qx)\,dH(Q)\,d\mu(x).
$$
Right invariance of <Haar measure> and transitivity of the <orthogonal group> on each <sphere> show that the inner average depends only on $r=|x|$. Averaging it once more over $x=r\theta$, using invariance of <surface measure on a sphere> and <Fubini's theorem>, identifies its value as $A\eta(r)$. At $x=0$ the same identity reads $\eta(0)=\eta(0)$. The definition of <pushforward measure> now proves the boxed formula. This argument also works for invariant <probability measures> with an <atom of a measure> at the origin.

For the printed normalization, choose a <smooth cutoff function> $\eta$ equal to one on a ball containing the <support of a measure> $\mu$. Its left integral is one, whereas the printed right integral is $\sigma_d$. Thus the extra $\sigma_d$ cannot be retained. Equivalently, with unnormalized area measure $dS$, the average is $\sigma_d^{-1}\int\eta(r\theta)\,dS(\theta)$.

\b[The reflection-averaging hint also fails in general dimension.] For $x=e_1$,
$$
\int_{\mathbb S^{d-1}}R_\omega e_1\,d\sigma(\omega)=\left(1-\frac2d\right)e_1,
\qquad\int_{\mathbb S^{d-1}}\omega\,d\sigma(\omega)=0.
$$
The first equality uses $\int\omega_i\omega_j\,d\sigma=\delta_{ij}/d$, which follows from coordinate reflections, coordinate permutations, and $\sum_i\omega_i^2=1$. Taking a <test function> equal to the first coordinate near the unit <sphere> disproves the hint when $d\ne2$. Averaging over the full <orthogonal group>, as above, proves the required corrected result without that hint.