Past exam of the mathematics course of the University of Cambridge / 2013 / iii / Paper 30 / 1

Use the normal linear modelHere $Y$ is the response random vector, $X$ is the known design matrix, $\beta \in {\mathbb{R}}^p$ contains the unknown regression coefficients, and ${\sigma }^2>0$ is the common error variance. Conditional on $X$, the errors have normal distributions and are independent random variables. Require $\mathrm{rank}X=p\le n$ for identifiability of $\beta$ and invertibility of $X^{T}X$. Usually $n>p$ is needed to estimate the error variance from the regression residuals. An intercept, when included, is represented by a column of ones in $X$.

Full column matrix rank makes $X^{T}X$ a positive-definite matrix, and its matrix inverse is symmetric. Thus the hat matrix satisfiesAlso $PX=X$ and the image of $P$ is contained in the column space of $X$. These identities show that $P$ is the orthogonal projection matrix onto the column space of $X$.

The ordinary least squares estimator is $\hat{\beta }=(X^{T}X{)}^{-1}{X}^{T}Y$. Consequently the fitted values and regression residuals areAn affine transformation of a multivariate normal distribution is again multivariate normal, possibly with a singular covariance matrix. For a random vector with covariance matrix $\Sigma$, its transformed covariance matrix is $A\Sigma A^T$. Since $PX=X$, $P^2=P=P^T$ and $(I-P{)}^2=I-P$, these results giveBoth multivariate normal distributions are supported on their respective projected subspaces. In particular, ${\hat{Y}}_i$ has variance ${\sigma }^2{P}_{ii}$ and $e_i$ has variance ${\sigma }^2(1-P_{ii})$; the regression residuals need not be mutually independent.

Because both vectors are linear transformations of the same multivariate normal response, $(\hat{Y},e)$ is jointly multivariate normal. Its cross-covariance matrix isZero cross-covariance implies independence for jointly multivariate normal vectors, including singular ones. Therefore the fitted values and the entire vector of regression residuals are independent. The fitted-residual orthogonality identity gives the zero covariance; the normal distribution assumption is what upgrades it to independence.

The simple linear regression assumes a straight conditional mean, $E(Y_i\mid x_i)={\beta }_0+{\beta }_1{x}_i$, so that errors are centred at zero throughout the predictor range. The displayed regression residuals are predominantly positive at both ends and negative in the middle. This is a residual curvature diagnostic: the fitted straight line misses a curved conditional mean. The main concern is the shape of the mean function. The plot alone does not establish failure of the normal distribution assumption or a particular error variance model.

A natural next fit is quadratic regression, which remains a normal linear model in its unknown regression coefficients:The new design matrix has rows $(1,x_i,x_i^2)$ and must have matrix rank three; three distinct predictor values suffice. The curved regression residual pattern suggests trying a positive quadratic term, but its sign and adequacy should be checked after fitting. Inspect the new regression residuals to see whether the systematic curvature has disappeared.