Past exam of the mathematics course of the University of Cambridge / 2023 / ii / Paper 1 / 13J (Statistical Modelling)

Writewhere $\epsilon$ and ${\epsilon }^{\ast }$ are independent, centered, and have covariance matrix ${\sigma }^2{I}_n$. ThenThe deterministic term is orthogonal in expectation to the two centered random terms, and the random terms are independent. Since the hat matrix is a symmetric idempotent projection of rank $p$,while $\mathbb{E}\Vert {\epsilon }^{\ast }{\Vert }^2=n{\sigma }^2$. ThereforeIn this bias-variance decomposition for linear prediction, $\Vert (I-H)\mu {\Vert }^2$ is squared model bias, $p{\sigma }^2$ is variance from fitting $p$ coefficients, and $n{\sigma }^2$ is irreducible noise in the future response. Enlarging the model tends to reduce the first term while increasing the fitted-model variance, which is the bias-variance tradeoff.

Because $I-H$ is an orthogonal projection of rank $n-p$,It follows that Mallows Cp satisfieswhere the last equality is part (i). Thus $C_p$ is an unbiased estimator of the independent-copy prediction error, as summarized by the unbiased prediction-error identity for ordinary least squares.

LetThe pivotal quadratic form has an $F_{p,n-p}$ distribution, so the $(1-\alpha )$ normal linear-model confidence ellipsoid isHere $F_{p,n-p}(1-\alpha )$ is the $(1-\alpha )$ quantile of the indicated $F$ distribution.

Let ${\hat{\beta }}_{(i)}$ be the least-squares estimate after deleting observation $i$, and put $\hat{Y}=X\hat{\beta }$ and ${\hat{Y}}_{(i)}=X{\hat{\beta }}_{(i)}$. Cook's distance isThus $D_i$ is the squared displacement caused by deleting observation $i$, measured in the same $X^{T}X$ metric and scale as the confidence ellipsoid. In particular, ${\hat{\beta }}_{(i)}$ lies outside the $(1-\alpha )$ ellipsoid centered at $\hat{\beta }$ exactly when

Here $p=4$ and $n-p=96$. The supplied quantile givesSincethe coefficient estimate after deleting this observation lies outside the $90\%$ confidence ellipsoid centered at the full-data estimate. It also greatly exceeds the common rough reference value $D_i=1$. The observation is therefore highly influential and merits investigation: