Past exam of the mathematics course of the University of Cambridge / 2011 / ii / Paper 4 / 13J / Solution

For ${\sigma }^2>0$, the Gaussian linear model has likelihood functionMaximizing over $\beta$ minimizes the residual sum of squares. Full column rank makes $X^{T}X$ positive definite, and differentiating gives the unique least-squares estimatorPut $P=X(X^{T}X{)}^{-1}{X}^T$ and $R=\Vert (I-P)Y{\Vert }^2$. If $R>0$, differentiating the profile log-likelihood in ${\sigma }^2$ givesThe denominator is $n$, not the residual degrees of freedom $n-p$; the latter gives the unbiased estimator.

A linear transformation of the multivariate normal distribution remains normal, soIts covariance with the residual vector is ${\sigma }^2(X^{T}X{)}^{-1}{X}^T(I-P)=0$. Jointly Gaussian vectors with zero cross-covariance are independent, hence $\hat{\beta }$ is independent of $(I-P)Y$, and therefore of ${\hat{\sigma }}^2$, a function of that residual. Orthogonal coordinates also give $R/{\sigma }^2\sim {\chi }_{n-p}^2$.

The allowed boundary case $p=n$ needs qualification. Then $P=I$, $R=0$ for every observation, and the likelihood increases without bound as ${\sigma }^2\downarrow 0$ after fitting $Y$ exactly. There is no maximum in the positive-variance model; $R/n=0$ is only a formal boundary estimate. When $n>p$, $R>0$ almost surely under a genuinely positive variance, and the displayed MLEs apply. A zero residual in any exceptional exact-fit sample has the same nonexistence issue. The estimator $\hat{\beta }$ and its normal distribution remain well-defined in the square full-rank case.