Past exam of the mathematics course of the University of Cambridge / 2007 / ib / Paper 4 / 19C (Statistics)

Write $S_{xx}={\sum }_i{x}_i^2$ and $\bar{Y}=n^{-1}{\sum }_i{Y}_i$. For an identifiable two-parameter linear regression assume $S_{xx}>0$; for the variance estimate and test below also assume $n>2$. The least-squares normal equations areUsing ${\sum }_i{x}_i=0$ givesThe residual sum of squares has positive-definite quadratic part $n{\alpha }^2+S_{xx}{\beta }^2$, so these equations give its unique minimum. The normal distribution of the independent errors gives the log-likelihoodFor every fixed positive ${\sigma }^2$, maximizing this over $\alpha ,\beta$ is exactly minimizing the same residual sum of squares. Thus the ordinary least squares estimators are also the maximum-likelihood estimators.

The printed hypotheses do not explicitly require $S_{xx}>0$ or $n>2$. If $S_{xx}=0$, all $x_i$ vanish and $\beta$ is unidentifiable; this is an intercept-only model with $n-1$ residual degrees of freedom. For $n=1$ its residual is identically zero, so the likelihood has no finite maximizer with positive variance. If $S_{xx}>0$ but $n=2$, the fit is saturated, leaving no residual variance estimate or Student test with positive degrees of freedom. The following nondegenerate formulas use $S_{xx}>0$ and $n>2$.

Let $\mathrm{RSS}={\sum }_i(Y_i-\hat{\alpha }-\hat{\beta }{x}_i{)}^2$. Differentiating the log-likelihood in $v={\sigma }^2$ givesFor $\mathrm{RSS}>0$ this changes from positive to negative at $v=\mathrm{RSS}/n$, soTo find its bias, substitute the model and use the orthogonal fit directions $1$ and $\mathbf{x}$, obtainingThe three expectations are $n{\sigma }^2$, ${\sigma }^2$, and ${\sigma }^2$, respectively: $\mathrm{Var}(\bar{\epsilon })={\sigma }^2/n$ and $\mathrm{Var}({\sum }_i{x}_i{\epsilon }_i)={\sigma }^2{S}_{xx}$. Hence $\mathbb{E}\mathrm{RSS}=(n-2){\sigma }^2$, giving the unbiased estimatorWith $n>2$, a zero residual sum of squares is a probability-zero event in the model. For that exceptional dataset the likelihood is unbounded as ${\sigma }^2\downarrow 0$, rather than having a maximum at a positive variance. In the intercept-only case $S_{xx}=0$, the corresponding unbiased multiplier is $n/(n-1)$ for $n>1$.

Set $\mathbf{e}=1/\sqrt{n}$ and $\mathbf{f}=\mathbf{x}/\sqrt{S_{xx}}$. These are orthonormal because ${\sum }_i{x}_i=0$. Complete them to an orthonormal basis $\mathbf{e},\mathbf{f},{\mathbf{q}}_3,\dots ,{\mathbf{q}}_n$ of ${\mathbb{R}}^n$. An orthogonal transformation of the standard normal error vector $\mathbf{\epsilon }/\sigma$ has independent normal random variables as coordinates $Z_1,\dots ,Z_n$: its joint probability density remains $(2\pi {)}^{-n/2}\exp (-\Vert z{\Vert }^2/2)$ because the transformation preserves squared length and has Jacobian determinant of absolute value one.

Projection onto these basis directions givesTherefore the joint distribution of least-squares and variance estimators isThe last law is the chi-squared distribution of a sum of $n-2$ squared independent normal random variables with mean zero and variance one, and independence follows because the three statistics use disjoint independent coordinate sets. When all $x_i=0$, omit the slope direction: $\hat{\alpha }$ and $n{\hat{\sigma }}^2/{\sigma }^2\sim {\chi }_{n-1}^2$ are independent, and there is no identifiable slope estimator.

Under the null hypothesis, the standardized intercept $Z=\sqrt{n}(\hat{\alpha }-{\alpha }_0)/\sigma$ is standard normal, and $V=\mathrm{RSS}/{\sigma }^2$ is independent with chi-squared distribution of $\nu =n-2$ degrees of freedom. Consequently the Student t test for a simple-regression intercept usesChoose a significance level $\gamma \in (0,1)$ before observing the test result. For the two-sided alternative, reject $H_0$ when $|T|>t_{n-2,1-\gamma /2}$, where the subscript denotes the indicated quantile of the Student t-distribution. The null rejection probability is exactly $\gamma$, independent of the nuisance slope and variance. Equivalently report the p-value $2[1-F_{t_{n-2}}(|T_{\mathrm{obs}}|)]$. The test assumes the normal independent-error model and a positive residual variance estimate. In the intercept-only case with $n>1$, use $s^2=\mathrm{RSS}/(n-1)$ and $t_{n-1}$ instead.