Past exam of the mathematics course of the University of Cambridge / 2022 / iii / Paper 218 / 5

For a model with $k$ fitted parameters and maximized likelihood $L(\hat{\theta })$, the Akaike information criterion isBackward selection starts with the full model, deletes the single variable giving the smallest AIC when that AIC is lower than the current value, and repeats until no deletion improves it. The first deletion is "indus", which gives AIC $1597.8$.

Adding variables can reduce training bias and increase the maximized likelihood, but it also increases estimation variance and optimism. The $2k$ penalty estimates this optimism, so minimizing AIC implements a bias-variance tradeoff aimed at expected out-of-sample Kullback–Leibler performance.

For $Y\sim N_n(X\beta ,{\sigma }^{2}I)$,

At the maximum-likelihood estimates,soThere are $p$ regression coefficients and one variance parameter. Adding twice this parameter count gives

The common terms cancel, andThis is negative exactly when

There are three regression coefficients and $97$ residual degrees of freedom, so $n=100$. The reported residual standard error uses the unbiased divisor:Hence the maximum-likelihood variance estimate is ${\hat{\sigma }}^2=\mathrm{RSS}/100$. Substituting this, $n=100$, and the four fitted parameters—three coefficients plus ${\sigma }^2$—into the formula in part c(ii) gives the AIC.