= Solution
The <ordinary least squares> estimate, which is also the <maximum-likelihood estimate> in this <normal linear model>, gives the <hat matrix>
$$
\boxed{P=X(X^TX)^{-1}X^T.}
$$
It is the <orthogonal projection> onto the column space of $X$, so $P^T=P$, $P^2=P$ and $\operatorname{rank}P=p$. Since $(I-P)X=0$, the vector of <regression residuals> is $\widehat\epsilon=(I-P)\epsilon$, with <multivariate normal distribution>
$$
\boxed{\widehat\epsilon\sim N_n\bigl(0,\sigma^2(I-P)\bigr).}
$$
For $i\ne j$, its <covariance> is $-\sigma^2p_{ij}$, which is generally nonzero. Therefore the <regression residuals> are generally dependent, even though the original errors are <independent random variables>.
Fit the model in R, compute its <internally studentized residuals> with `rstandard(fit)`, and examine a <normal Q-Q plot> with `qqnorm(rstandard(fit))` and `qqline(rstandard(fit))`. A plot of these <regression residuals> against the <fitted values> helps detect a curved mean pattern or nonconstant <variance>. Approximate agreement with a straight line in the <Q-Q plot>, with no systematic residual pattern, supports the assumed error model.
Here $p\ll n$. The average <leverage> is $\operatorname{tr}P/n=p/n$, and $\sum_{ij}p_{ij}^2=\operatorname{tr}(P^2)=p$. Thus most diagonal corrections and most off-diagonal dependences are small when only a few directions have been removed from a large sample. Also $\widetilde\sigma^2$ estimates $\sigma^2$ accurately when $n-p$ is large. These facts make the visual independent-normal approximation reasonable, though exceptional high-<leverage> observations still deserve attention.
The approximation is not an exact distributional statement: the <distribution of an internally studentized Gaussian residual> has $\widehat\eta_i^2/(n-p)\sim\operatorname{Beta}(1/2,(n-p-1)/2)$ when $n-p\geq2$ and $p_{ii}<1$. In particular these residuals are bounded, rather than exactly distributed according to <Student's t-distribution>. Formal goodness-of-fit calibration should account for fitting and dependence.
Back to article page
