Past exam of the mathematics course of the University of Cambridge / 2017 / ib / Paper 3 / 20H (Statistics)

The Gauss-Markov theorem states that with full-column-rank design, mean-zero errors and covariance ${\sigma }^{2}I$, the ordinary least squares estimator is the best linear unbiased estimator: its covariance is no larger than any other such estimator's in the Loewner order. Normality is unnecessary for that theorem.

In the given normal linear model, maximizing the likelihood function is equivalent to minimizing $\Vert Y-X\beta {\Vert }^2$. The normal equations and full rank giveFor a direct variance comparison, write ${\beta }^{\ast }=AY$. Unbiasedness for every $\beta$ requires $AX=I_p$. Set $B=(X^{T}X{)}^{-1}{X}^T$ and $D=A-B$, so $DX=0$. The cross covariance terms vanish because $B{D}^T=(X^{T}X{)}^{-1}(DX{)}^T=0$. ThereforeThis proves the requested comparison for every contrast $t$ without relying only on the theorem's name.

Write the one-column design as $x\in {\mathbb{R}}^n\setminus \{0\}$ and $S=x^{T}x$. Maximizing the normal linear model likelihood first gives $\hat{\beta }=x^{T}Y/S$. Its minimized residual sum of squares is $R=\Vert Y-x\hat{\beta }{\Vert }^2$. For variance $v>0$, the remaining log likelihood is $-\frac{n}{2}\log v-R/(2v)$ plus a constant, whose maximum is atHere $R>0$ almost surely because $n>1$ and the true variance is positive. In the exceptional data case $R=0$, the likelihood is unbounded as $v\downarrow 0$, so there is no positive-variance maximum.

Let $P=x{x}^T/S$, the orthogonal projection matrix onto the design column. Then $\hat{\beta }-\beta =x^T\epsilon /S$ and $Y-x\hat{\beta }=(I-P)\epsilon$. These are orthogonal linear projections of an isotropic multivariate normal distribution, hence independent. An orthonormal coordinate system adapted to $x$ makes the residual squared length a sum of $n-1$ independent squared standard normals. Thus the joint distribution of least-squares and variance estimators is the product lawIn particular the maximum likelihood variance estimator is biased: $\mathbb{E}{\hat{\sigma }}^2=(n-1){\sigma }^2/n$; the unbiased residual estimator uses denominator $n-1$.