Past exam of the options II course of the University of Oxford / 2011 / O / O5

The Gaussian linear model has log likelihood function, up to an additive constant,For each fixed positive ${\sigma }^2$, maximize this by minimizing the residual sum of squares. Differentiation gives the normal equationsThe full column rank of $X$ makes $X^{T}X$ positive definite, hence invertible, soThe quadratic objective is strictly convex, which proves uniqueness. The joint maximum-likelihood estimate has the same coefficient estimate, except on the probability-zero event of an exactly zero residual when the positive-variance likelihood has no finite joint maximizer.

Write $A=(X^{T}X{)}^{-1}{X}^T$. Since $AX=I_p$,The errors have mean zero and covariance matrix ${\sigma }^2{I}_n$. ThereforeThus $\hat{\theta }$ is an unbiased estimator. In addition, the multivariate normal distribution is preserved by linear maps, so $\hat{\theta }$ is multivariate normal with the displayed mean and covariance matrix.

Use the residual mean squareTo justify its denominator, let $H=X(X^{T}X{)}^{-1}{X}^T$, the orthogonal projection matrix onto the column space of $X$. Then the residual is $(I-H)ϵ$. The idempotent matrix $I-H$ has rank $n-p$, so diagonalizing it with an orthogonal matrix givesConsequently $\mathbb{E}{s}^2={\sigma }^2$, making $s^2$ an unbiased estimator. If specifically asked for maximum likelihood estimation, maximizing over ${\sigma }^2$ gives the alternative ${\hat{\sigma }}_{\mathrm{ML}}^2=\Vert Y-X\hat{\theta }{\Vert }^2/n$, which is biased downward by the factor $(n-p)/n$.

Let $v_i=[(X^{T}X{)}^{-1}{]}_{ii}>0$. Part (b) givesThe residual vector and coefficient estimator are independent: they are jointly normal and their cross-covariance matrix is zero, since $X^T(I-H)=0$. Hence $Z_i$ is independent of $U=(n-p)s^2/{\sigma }^2\sim {\chi }_{n-p}^2$. By the definition of Student's t-distribution,Let $t_{n-p,1-\alpha /2}$ be the $1-\alpha /2$ quantile of this distribution. Inverting the central probability statement gives the exact marginal confidence intervalIts coverage is $1-\alpha$ under the stated Gaussian linear model; simultaneous coverage for all coefficients would require a separate adjustment.