Past exam of the options II course of the University of Oxford / 2009 / O / O4

Minimizing the sum of squared residuals gives the normal equations for linear least squares, $X^{T}X\hat{\theta }=X^{T}Y$. Full column rank makes $X^{T}X$ positive definite and invertible, so the unique minimizer isBecause $\mathbb{E}Y=X\theta$ and $\mathrm{Cov}(Y)={\sigma }^2{I}_n$, one has $AX=I_p$ andFor the alternative linear estimator the corresponding general formulas areNo normality assumption on the errors is needed for these claims.

Unbiasedness of an estimator means unbiasedness for every parameter value, not just one chosen $\theta$. Thus $DX\theta =\theta$ for every $\theta \in {\mathbb{R}}^p$. Evaluating this identity on a basis, or identifying the two linear maps, gives

Put $D^{\ast }=D-A$, with $A=(X^{T}X{)}^{-1}{X}^T$. Then $D^{\ast }X=0$, and therefore $D^{\ast }{A}^T=D^{\ast }X(X^{T}X{)}^{-1}=0$; its transpose also vanishes. Expanding givesHenceThe difference from the least-squares covariance is positive semidefinite, since $v^T{D}^{\ast }{D}^{\ast T}v=\Vert D^{\ast T}v{\Vert }^2\ge 0$. In particular its $i$th diagonal entry is ${\sigma }^2{\sum }_j(D_{ij}^{\ast }{)}^2$, proving every stated variance inequality. For estimating ${\theta }_1$, prefer least squares: among linear unbiased estimators it has minimum variance, and therefore minimum mean squared error in that class. A competitor can tie if the first row of $D^{\ast }$ is zero. This is the Gauss-Markov theorem; it does not assert optimality against all biased or nonlinear estimators.