Past exam of the mathematics course of the University of Oxford / Undated / Prelims / Mathematics III resit / 7

The least-squares estimators choose the intercept and slope to minimize the sum of squared regression residuals:After observation, replace $Y_i$ by $y_i$ to obtain the numerical estimates. Their definition does not require a normal distribution for the errors. A unique slope requires some variation in the fixed predictors; under ${\sum }_i{x}_i=0$, this is $S_{xx}={\sum }_i{x}_i^2>0$, assumed below. If all $x_i=0$, the slope is unidentifiable.

Differentiate the ordinary least squares objective in a. Its normal equations for linear least squares areUsing ${\sum }_i{x}_i=0$ and $S_{xx}={\sum }_i{x}_i^2>0$ givesThe Hessian matrix of the objective in $(a,b)$ is $2\mathrm{diag}(n,S_{xx})$, which is positive-definite, so this is its unique global minimum. For the observed estimates, replace $Y_i$ by $y_i$.

The centered design gives $\hat{\beta }=\beta +S_{xx}^{-1}{\sum }_i{x}_i{ϵ}_i$. By independence of the errors, their off-diagonal covariances vanish. Thus variance additivity for independent random variables givesA larger spread of predictor values decreases the slope variance for fixed error variance.

Here $\hat{\alpha }=\alpha +n^{-1}{\sum }_i{ϵ}_i$. By linearity of covariance and independence,Thus the two estimators are uncorrelated random variables.

They need not be independent. For a counterexample take $n=2$, $x_1=-1$, $x_2=1$, and independent errors taking $-1,1$ with equal probability. ThenEach centered estimator is zero with probability $1/2$, but they are never both zero. Their zero events therefore violate independence.

A sufficient additional assumption is that the errors have a common normal distribution. The two estimators are then Jointly Gaussian, being linear combinations of the independent Gaussian random variables. Their zero covariance, established in iii, implies independence of uncorrelated jointly normal variables. Normal errors are a sufficient condition here, not a claim that no other error probability distribution could ever work.

For the random vector $\mathbf{ϵ}=({ϵ}_1,\dots ,{ϵ}_n{)}^{\top }$, its covariance matrix isIts diagonal entries are variances, and its off-diagonal entries measure pairwise covariance. Finite second moments make every entry well defined.

Independence of the centered standard normal random variables gives $\mathbb{E}{ϵ}_i=0$ for every $i$. AlsoFor nonadjacent indices, the underlying sets of $\eta$ variables are disjoint, so the errors are independent and have zero covariance. For adjacent indices with $i\ge 2$,The first adjacent pair separately gives $\mathbb{E}[{ϵ}_1{ϵ}_2]=\mathbb{E}[{\eta }_1^2{\eta }_2]=0$. HenceThese adjacent products of independent Gaussian variables are uncorrelated random variables even though adjacent errors share a factor and are dependent.

The least-squares estimators depend only on the observed responses and design, so the formulas in i still hold. Their variances and covariance need only the error covariance matrix, now $I_n$. ThereforeFor example the first formula follows from $\mathrm{Var}({\sum }_i{x}_i{ϵ}_i)={\sum }_i{x}_i^2$, and the second from ${\sum }_i{x}_i=0$. The formulas remain the same as before with ${\sigma }^2=1$; this calculation does not establish independence or a normal distribution for the estimators.

Compute fitted responses ${\hat{y}}_i=\hat{\alpha }+\hat{\beta }{x}_i$ and regression residuals $e_i=y_i-{\hat{y}}_i$. A conventional residual plot shows $e_i$ against ${\hat{y}}_i$ or $x_i$; plotting against observation order additionally looks for runs. One checks for curvature, changing spread and other structure rather than random scatter around zero.

An ordinary residual plot need not reveal this failure of independence. The errors have constant mean zero, constant variance one and zero pairwise covariances, so the usual trend and changing-variance clues need not appear. A visually patternless residual plot cannot validate independence; fitted regression residuals themselves also share the fitted coefficients even when the true errors are independent.

There is detectable dependence in the magnitudes. The Gaussian fourth moment is $\mathbb{E}{\eta }_i^4=3$, giving for every adjacent pairFor $i=1$, the product is ${\eta }_1^4{\eta }_2^2$; for $i\ge 2$, it is ${\eta }_{i-1}^2{\eta }_i^4{\eta }_{i+1}^2$, with the same expected value 3. Large magnitudes tend to occur together. A lagged plot of squared regression residuals or a test directed at dependence in their squares is more informative. Thus recognition is possible with suitable diagnostics, but is not guaranteed by the standard residual plot requested here.