 Solution

Put $D^{\ast }=D-A$, with $A=(X^{T}X{)}^{-1}{X}^T$. Then $D^{\ast }X=0$, and therefore $D^{\ast }{A}^T=D^{\ast }X(X^{T}X{)}^{-1}=0$; its transpose also vanishes. Expanding givesHenceThe difference from the least-squares covariance is positive semidefinite, since $v^T{D}^{\ast }{D}^{\ast T}v=\Vert D^{\ast T}v{\Vert }^2\ge 0$. In particular its $i$th diagonal entry is ${\sigma }^2{\sum }_j(D_{ij}^{\ast }{)}^2$, proving every stated variance inequality. For estimating ${\theta }_1$, prefer least squares: among linear unbiased estimators it has minimum variance, and therefore minimum mean squared error in that class. A competitor can tie if the first row of $D^{\ast }$ is zero. This is the Gauss-Markov theorem; it does not assert optimality against all biased or nonlinear estimators.