Past exam of the mathematics course of the University of Cambridge / 2016 / iii / Paper 206 / 2 / b

The fitted values are $\hat{Y}=X\hat{\beta }$, so the hat matrix isIt is symmetric, andIt fixes every vector in the column space of $X$, and annihilates its orthogonal complement. Thus $H$ is the orthogonal projection onto that column space, has matrix rank and trace $p$, and $I-H$ is the complementary orthogonal projection. ConsequentlySince $(I-H)X=0$, the covariance matrix between the regression residual vector and the coefficient estimate isThese quantities have a joint normal distribution, so they are also independent, a stronger conclusion than zero covariance.

Now $\hat{e}=(I-H)\epsilon$ has mean zero. Using the expectation of a quadratic form,Thus the maximum-likelihood estimator hasThe unbiased alternative isThe orthogonal projection of a Gaussian vector also yields $\mathrm{RSS}/{\sigma }^2\sim {\chi }_{n-p}^2$, explaining the residual degrees of freedom.