Past exam of the mathematics course of the University of Cambridge / 2024 / ii / Paper 4 / 30K / c / Solution

The matrixis the orthogonal projection onto the row space of $X$. Since $XP\beta =X\beta$, the empirical risks of $h_{\beta }$ and $h_{P\beta }$ are equal. Orthogonality givesso $q(\beta )\ge q(P\beta )$, with strict inequality unless $(I-P)\beta =0$. Therefore the minimizer lies in the row space:Because $X^T$ is injective when $X{X}^T$ is invertible, minimizing over that row space is equivalent to minimizing $r(\alpha )=q(X^T\alpha )$.