= Solution
The <matrix>
$$
P=X^T(XX^T)^{-1}X
$$
is the orthogonal projection onto the row space of $X$. Since $XP\beta=X\beta$, the empirical risks of $h_\beta$ and $h_{P\beta}$ are equal. Orthogonality gives
$$
\|\beta\|^2=\|P\beta\|^2+\|(I-P)\beta\|^2,
$$
so $q(\beta)\geq q(P\beta)$, with strict inequality unless $(I-P)\beta=0$. Therefore the minimizer lies in the row space:
$$
\widehat\beta=P\widehat\beta=X^T\widehat\alpha.
$$
Because $X^T$ is injective when $XX^T$ is invertible, minimizing over that row space is equivalent to minimizing $r(\alpha)=q(X^T\alpha)$.
Back to article page
