Past exam of the mathematics course of the University of Oxford / 2017 / Prelims / Mathematics III / 8

Treat the explanatory variables as fixed and put $S_{xx}={\sum }_i{x}_i^2$, $S_{zz}={\sum }_i{z}_i^2$. To identify both slopes, assume $S_{xx}>0$ and $S_{zz}>0$. The given centring and orthogonality conditions make the three columns $1,x,z$ pairwise orthogonal.

The log-likelihood function in the normal linear model, up to a constant independent of the coefficients, isThus maximum likelihood estimation minimizes the residual sum of squares. Setting its partial derivatives to zero givesUsing $\sum x_i=\sum z_i=\sum x_i{z}_i=0$ decouples these equations:The Hessian matrix of the residual sum of squares is $2\mathrm{diag}(n,S_{xx},S_{zz})$, which is positive definite. These are therefore the unique minimizers. If either squared sum is zero, the associated explanatory variable vanishes identically and its coefficient cannot be identified.

Substitute the linear regression model into the estimators and use the centring and orthogonality conditions:A linear combination of independent normal random variables is normal. Their means are ${\beta }_0$ and ${\beta }_1$, while variance additivity for independent random variables givesConsequently

The central limit theorem states that if $W_1,W_2,\dots$ are independent and identically distributed with finite mean $\mu$ and variance $0<{\tau }^2<\infty$, thenHere $\stackrel{d}{\to }$ denotes convergence in distribution.

The algebraic identity ${\hat{\beta }}_0-{\beta }_0=n^{-1}{\sum }_i{ϵ}_i$ does not require a normal distribution for the errors. Applying the central limit theorem to their sample mean givesThus, for large $n$,Its mean ${\beta }_0$ and variance ${\sigma }^2/n$ remain exact, even though its normal distribution is now only an approximation. With the error probability distribution unspecified, the formula is retained as an estimator; it is not being asserted to remain a maximum likelihood estimation formula.

No, not from the stated assumptions alone. The centered slope is a weighted sum,and the weighted errors need not be identically distributed. The ordinary central limit theorem therefore does not apply directly.

For a concrete counterexample, for every $n\ge 3$ chooseThese satisfy all three centring and orthogonality assumptions, and both explanatory columns are nonzero. Let the errors be independent with Rademacher distribution, taking values $\pm 1$ with equal probability. Thenwhich takes values $-1,0,1$ with probabilities $1/4,1/2,1/4$ for every $n$. Increasing the sample size adds no information about this slope, and the standardized estimator does not approach a normal distribution.

A useful additional assumption for asymptotic normality of a weighted regression slope is that no single design value dominates:Under this condition and the given finite error variance, the Lindeberg-Feller central limit theorem yieldsjustifying the approximation $N({\beta }_1,{\sigma }^2/S_{xx})$ for the slope as well.