= Solution
Put $D^*=D-A$, with $A=(X^TX)^{-1}X^T$. Then $D^*X=0$, and therefore $D^*A^T=D^*X(X^TX)^{-1}=0$; its transpose also vanishes. Expanding gives
$$
DD^T=(A+D^*)(A+D^*)^T=(X^TX)^{-1}+D^*D^{*T}.
$$
Hence
$$
\boxed{\operatorname{Cov}(\widetilde\theta)=\sigma^2\left((X^TX)^{-1}+D^*D^{*T}\right)}.
$$
The difference from the least-squares covariance is positive semidefinite, since $v^TD^*D^{*T}v=\|D^{*T}v\|^2\geq0$. In particular its $i$th diagonal entry is $\sigma^2\sum_j(D^*_{ij})^2$, proving every stated <variance> inequality. For estimating $\theta_1$, \b[prefer least squares]: among linear unbiased estimators it has minimum <variance>, and therefore minimum mean squared error in that class. A competitor can tie if the first row of $D^*$ is zero. This is the <Gauss-Markov theorem>; it does not assert optimality against all biased or nonlinear estimators.
Back to article page
