Past exam of the Probability, statistics and options II course of the University of Oxford / 2009 / O / O4

Minimizing $\Vert Y-X\theta {\Vert }^2$ gives the normal equations $X^{T}X\hat{\theta }=X^{T}Y$. Since $X$ has full column rank, $X^{T}X$ is positive definite and invertible, and the objective has a unique minimum. Set $A=(X^{T}X{)}^{-1}{X}^T$; then $\hat{\theta }=AY$ and $AX=I_p$. The model gives $\mathbb{E}[Y]=X\theta$ and $\mathrm{Var}(Y)={\sigma }^2{I}_n$, so $\mathbb{E}[\hat{\theta }]=\theta$ and $\mathrm{Var}(\hat{\theta })={\sigma }^{2}A{A}^T={\sigma }^2(X^{T}X{)}^{-1}$. For the alternative estimator the corresponding general formulas are $\mathbb{E}[\tilde{\theta }]=DX\theta$ and $\mathrm{Var}(\tilde{\theta })={\sigma }^{2}D{D}^T$.

Unbiasedness as an estimator of the parameter means $DX\theta =\theta$ for every $\theta \in {\mathbb{R}}^p$, rather than just one particular value. Thus $(DX-I_p)\theta =0$ for every vector, forcing $\boxed{DX=I_p}$. Conversely that identity plainly suffices for unbiasedness.

Put $D^{\ast }=D-A$. By (i), $D^{\ast }X=0$. Thus $D^{\ast }{A}^T=D^{\ast }X(X^{T}X{)}^{-1}=0$, and its transpose vanishes too. Expanding $D{D}^T=(D^{\ast }+A)(D^{\ast }+A{)}^T$ yieldsThe covariance difference is positive semidefinite. In particular its $i$th diagonal entry is ${\sigma }^2{\sum }_j(D_{ij}^{\ast }{)}^2\ge 0$, proving the requested componentwise variance inequalities. Prefer $\boxed{{\hat{\theta }}_1}$ within the class of linear unbiased estimators, since it has the least variance and the same expectation. A tie is possible if the first row of $D^{\ast }$ is zero. This is the Gauss-Markov theorem conclusion; no normal-distribution assumption on the errors is required.