Past exam of the mathematics course of the University of Cambridge / 2008 / ii / Paper 3 / 5J / Solution

The ordinary least squares estimate, which is also the maximum-likelihood estimate in this normal linear model, gives the hat matrixIt is the orthogonal projection onto the column space of $X$, so $P^T=P$, $P^2=P$ and $\mathrm{rank}P=p$. Since $(I-P)X=0$, the vector of regression residuals is $\widehat{ϵ}=(I-P)ϵ$, with multivariate normal distributionFor $i\ne j$, its covariance is $-{\sigma }^2{p}_{ij}$, which is generally nonzero. Therefore the regression residuals are generally dependent, even though the original errors are independent random variables.

Fit the model in R, compute its internally studentized residuals with rstandard(fit), and examine a normal Q-Q plot with qqnorm(rstandard(fit)) and qqline(rstandard(fit)). A plot of these regression residuals against the fitted values helps detect a curved mean pattern or nonconstant variance. Approximate agreement with a straight line in the Q-Q plot, with no systematic residual pattern, supports the assumed error model.

Here $p\ll n$. The average leverage is $\mathrm{tr}P/n=p/n$, and ${\sum }_{ij}{p}_{ij}^2=\mathrm{tr}(P^2)=p$. Thus most diagonal corrections and most off-diagonal dependences are small when only a few directions have been removed from a large sample. Also ${\tilde{\sigma }}^2$ estimates ${\sigma }^2$ accurately when $n-p$ is large. These facts make the visual independent-normal approximation reasonable, though exceptional high-leverage observations still deserve attention.

The approximation is not an exact distributional statement: the distribution of an internally studentized Gaussian residual has ${\hat{\eta }}_i^2/(n-p)\sim \mathrm{Beta}(1/2,(n-p-1)/2)$ when $n-p\ge 2$ and $p_{ii}<1$. In particular these residuals are bounded, rather than exactly distributed according to Student's t-distribution. Formal goodness-of-fit calibration should account for fitting and dependence.