Past exam of the mathematics course of the University of Cambridge / 2005 / iii / Paper 41 / 3

Expanding the ordinary least squares objective givesFor a nonzero vector $v$, $v^T{X}^{T}Xv=\Vert Xv{\Vert }^2>0$, since the design matrix has matrix rank $p$. The Hessian matrix is therefore positive-definite, and $Q$ is strictly convex. Solving its normal equations gives the unique least-squares estimator:Indeed, expansion about this stationary point and $X^T(Y-X\hat{\beta })=0$ give $Q(\beta )=Q(\hat{\beta })+(\beta -\hat{\beta }{)}^T{X}^{T}X(\beta -\hat{\beta })$, which also proves the global minimum directly.

Define the hat matrix $H=X(X^{T}X{)}^{-1}{X}^T$. It is a symmetric matrix and an idempotent linear map, since $H^T=H$ and $H^2=H$, and its image is the column space of $X$. Thus $H$ is the orthogonal projection onto the fitted mean space. The fitted values are $HY$ and the regression residual vector is $(I-H)Y$. Because $I-H$ is also a symmetric matrix and an idempotent linear map,These minimization and projection identities do not need errors with a normal distribution; the normal linear model becomes relevant to the distributional calculations.

Put $G=I-H$. Since $GX=0$, the regression residual vector is $\hat{\epsilon }=G\epsilon$. A linear transformation of a multivariate normal distribution is again multivariate normal, soThis is a singular multivariate normal distribution supported on $\ker X^T$, with dimension $n-p$. In particular,Consequently regression residuals are neither independent nor identically distributed even under the stated model. Their linear constraints include $X^T\hat{\epsilon }=0$, and, if the design matrix contains an intercept, their sum is zero.

Choose an orthonormal basis of the residual space. The coordinates of $G\epsilon$ in that basis are $n-p$ independent $N(0,{\sigma }^2)$ variables, provingFor $n>p$, $s^2$ is an unbiased estimator of ${\sigma }^2$. Orthogonality of $H$ and $G$ gives zero cross-covariance, so the Gaussian fitted and residual vectors are independent. This explains why an appropriate residual-versus-fitted diagnostic has no systematic mean relationship under the model. If $n=p$, no residual information remains for these diagnostics or the variance estimate.

For $h_{ii}<1$, use standardized regression residuals $r_i={\hat{\epsilon }}_i/[s\sqrt{1-h_{ii}}]$ to account for the different residual variances. The diagonal $h_{ii}$ is the regression leverage; $h_{ii}=1$ makes that residual identically zero and cannot be diagnosed by this standardization. With known $\sigma$, the corresponding individually standardized residual has an exact standard normal distribution. With estimated $s$, the internally studentized residual is not exactly normal or a Student t-distribution, and the residuals remain correlated; a normal reference plot is a diagnostic approximation.

Plot the regression residuals or standardized regression residuals against fitted values and each covariate. A persistent curve suggests a misspecified mean, while a fan-shaped spread suggests heteroscedasticity; a scale-location plot helps separate these. A quantile-quantile plot against normal quantiles checks tails, skewness and possible regression outliers. Plot residuals against observation order or time to look for unexplained serial patterns, recognizing that projection already induces the displayed residual covariances. Inspect regression leverage together with Cook's distance for influential observations. Such patterns motivate revising the model; absence of a visible pattern does not prove all its assumptions.

The response errors have standard deviations proportional to $|{\mu }_i|$. Thus, when fitted means approximate the true means, increasing absolute fitted values tends to accompany increasing residual spread. A residual-versus-fitted plot shows a fan for positive means, or spread increasing away from zero for means of either sign. Standardization by one common residual standard deviation, even with the usual regression leverage correction, does not remove this systematic heteroscedasticity.

Under independent errors the exact covariance matrix after fitting iswhere $k>0$ is the common proportionality constant. Hence the simple fan description is a diagnostic tendency, not an assertion that each fitted residual has exactly variance $k{\mu }_i^2$.

For positive responses, the intended variance-stabilizing transformation is the natural logarithm. The delta method givesUse $\log Y_i$ for positive data with modest relative error, then refit and check the transformed mean and residual structure. In particular, if the original mean was $X_i^T\beta$, the transformed mean is approximately $\log (X_i^T\beta )-k/2$, not automatically a linear function of the original covariates.

There is a domain qualification in the literal printed model: a nondegenerate normal distribution has negative support. For positive ${\mu }_i$, the probability of $Y_i\le 0$ is $\Phi (-1/\sqrt{k})$, so a real $\log Y_i$ is not defined throughout the stated sampling model. When $k$ is small this is a negligible-tail approximation, not an exact transformation to normal errors. If signed values must be retained and all ${\mu }_i\ne 0$, write $Y_i={\mu }_i(1+\sqrt{k}{Z}_i)$ with standard normal $Z_i$. ThenThe second term has a common distribution with finite variance, so log absolute value stabilization of a normal scale family is exact, but loses the sign of the mean and does not produce normal errors. Zeros and a zero mean are excluded from this transform. Alternatively retain the response scale and fit the stated variance model with weighted least squares weights proportional to $1/{\hat{\mu }}_i^2$ where the means stay away from zero. No arbitrary addition of a constant before taking logarithms exactly solves the given variance relation.