Past exam of the mathematics course of the University of Cambridge / 2006 / ii / Paper 4 / 13I (Statistical Modelling)

The Gaussian log likelihood function, apart from a constant, isDifferentiating in $\beta$ gives $X^{\mathsf{T}}(Y-X\hat{\beta })=0$. Full column rank makes $X^{\mathsf{T}}X$ invertible, soThe distribution follows by applying this linear map to the Gaussian error vector. Differentiating the profiled likelihood function in ${\sigma }^2$ givesFor its distribution, $P=X(X^{\mathsf{T}}X{)}^{-1}{X}^{\mathsf{T}}$ is symmetric idempotent, and the residual is $(I-P)ϵ$. The needed form of Cochran's theorem says that quadratic forms in a standard Gaussian vector associated with orthogonal symmetric projection matrices summing to identity are independent random variables with chi-squared distributions with degrees equal to their ranks. Applying it to $P$ and $I-P$ givesThus the maximum-likelihood variance estimate uses divisor $n$ and has expectation $(n-p){\sigma }^2/n$; the unbiased residual estimate would use divisor $n-p$. The fitted coefficient and residual quadratic form are also independent.