Past exam of the mathematics course of the University of Cambridge / 2005 / ii / Paper 3 / 5I (Statistical Modelling)

The least-squares estimator minimizes $\Vert Y-Xb{\Vert }^2$. Differentiating in $b$ gives $-2X^T(Y-Xb)=0$, hence $\boxed{X^{T}X\hat{\beta }=X^{T}Y}$. Full column rank makes $X^{T}X$ positive definite, soThis is a linear transformation of a multivariate normal distribution. Its mean is $\beta$ and its covariance is ${\sigma }^2(X^{T}X{)}^{-1}$. Therefore $\boxed{\hat{\beta }\sim N_p(\beta ,{\sigma }^2(X^{T}X{)}^{-1})}$.

Define the hat matrix $H=X(X^{T}X{)}^{-1}{X}^T$. It is symmetric and idempotent, the orthogonal projection onto the column space of $X$. Since $HX=X$,The covariance is ${\sigma }^{2}H{H}^T={\sigma }^{2}H$. This normal distribution can be singular when $p<n$, since all fitted vectors lie in the $p$-dimensional model subspace.