Past exam of the mathematics course of the University of Cambridge / 2019 / iii / Paper 218 / 2

For a model with maximized log likelihood $\ell (\hat{\theta })$, $k$ fitted parameters, and $n$ observations,Both combine goodness of fit with a complexity penalty; smaller values are preferred.

Both criteria are used for model selection. AIC estimates relative out-of-sample predictive or Kullback-Leibler risk and is attractive when prediction is the main aim. BIC approximates a log Bayes factor under regular fixed-dimensional models and is consistent for selecting a true finite-dimensional model when one is present. Their penalties differ by $2k$ versus $k\log n$. For $n>e^2$, BIC penalizes each additional parameter more strongly and therefore tends to select smaller models.

For model $k$, let $X_k$ contain its $p_k$ active columns. With known noise variance one,and full column rank gives

Let $P_k$ be orthogonal projection onto the column space of $X_k$, and let $\mu$ be the true mean. ThenUp to a common constant,The true model has $p_k=p^{\ast }$ and ${\lambda }_k=0$; every other candidate has $p_k>p^{\ast }$ and ${\lambda }_k\ge 0$. Its excess expected AIC is ${\lambda }_k+p_k-p^{\ast }>0$, while its excess expected BIC is ${\lambda }_k+(\log n-1)(p_k-p^{\ast })>0$ when $n>e$. Thus either minimum expected criterion selects the true model.

With one true regressor and one additional candidate regressor, the reduction in residual sum of squares from fitting the larger nested model is $Z^2\sim {\chi }_1^2$. AIC chooses the wrong larger model exactly whenwhose probability is fixed and positive, independent of $n$. BIC chooses it whenFor $n>e^2$ this probability is strictly smaller than the AIC error probability, and it tends to zero as $n\to \infty$.