Past exam of the mathematics course of the University of Oxford / Undated / Prelims / Mathematics III resit / 7 / b

Differentiate the ordinary least squares objective in a. Its normal equations for linear least squares areUsing ${\sum }_i{x}_i=0$ and $S_{xx}={\sum }_i{x}_i^2>0$ givesThe Hessian matrix of the objective in $(a,b)$ is $2\mathrm{diag}(n,S_{xx})$, which is positive-definite, so this is its unique global minimum. For the observed estimates, replace $Y_i$ by $y_i$.

The centered design gives $\hat{\beta }=\beta +S_{xx}^{-1}{\sum }_i{x}_i{ϵ}_i$. By independence of the errors, their off-diagonal covariances vanish. Thus variance additivity for independent random variables givesA larger spread of predictor values decreases the slope variance for fixed error variance.

Here $\hat{\alpha }=\alpha +n^{-1}{\sum }_i{ϵ}_i$. By linearity of covariance and independence,Thus the two estimators are uncorrelated random variables.

They need not be independent. For a counterexample take $n=2$, $x_1=-1$, $x_2=1$, and independent errors taking $-1,1$ with equal probability. ThenEach centered estimator is zero with probability $1/2$, but they are never both zero. Their zero events therefore violate independence.

A sufficient additional assumption is that the errors have a common normal distribution. The two estimators are then Jointly Gaussian, being linear combinations of the independent Gaussian random variables. Their zero covariance, established in iii, implies independence of uncorrelated jointly normal variables. Normal errors are a sufficient condition here, not a claim that no other error probability distribution could ever work.