Past exam of the mathematics course of the University of Cambridge / 2020 / ib / Paper 2 / 18H (Statistics)

Because $X$ has full column rank, $X^{T}X$ is invertible. The ordinary least squares estimator isIt satisfies the normal equationsso the residual is orthogonal to the column space of $X$. For any $\beta$,The two terms are orthogonal, and the Pythagorean theorem in an inner-product space givesThus $\hat{\beta }$ minimizes the least-squares objective.

Since $\hat{\beta }={\beta }^0+(X^{T}X{)}^{-1}{X}^T\epsilon$ and $\mathrm{cov}(\epsilon )={\sigma }^2{I}_n$,

Write $\tilde{\beta }=({\tilde{\gamma }}^T,0{)}^T$, whereThe final coordinate is deterministic, while the first $p-1$ coordinates have the usual least-squares covariance. Hence

Because $\mathrm{col}Z\subseteq \mathrm{col}X$, the two orthogonal projection matrices satisfyConsequently $P-P_0$ is itself an orthogonal projection, onto the orthogonal complement of $\mathrm{col}Z$ within $\mathrm{col}X$. In particular it is a positive semidefinite matrix, so for every $u\in {\mathbb{R}}^n$,

Full column rank of $X$ makes $X^T:{\mathbb{R}}^n\to {\mathbb{R}}^p$ surjective. Given $x\in {\mathbb{R}}^p$, choose $u$ with $x=X^{T}u$. The fitted values for the unrestricted and restricted regressions areThereforeUsing $\mathrm{cov}(Y)={\sigma }^2{I}_n$ and idempotence of the projections,Part (d) now gives