Model selection

Model selection compares candidate statistical models and chooses one according to a criterion balancing fit, predictive performance and complexity.

For Bayesian deviance $D(\theta )=-2\log L(\theta )$ under a common likelihood convention, define $\overline{D}=\mathbb{E}[D(\theta )\mid y]$ and $p_D=\overline{D}-D(\mathbb{E}[\theta \mid y])$. Then $\mathrm{DIC}=\overline{D}+p_D$ trades average fit against effective complexity. Lower values suggest better penalized predictive fit in comparable models; differences are not Bayes factors or model posterior probabilities. The choice of stochastic parameters and latent-variable representation matters, especially in a hierarchical Bayesian model.

The DIC effective parameter count compares mean Bayesian deviance with deviance at the posterior mean. It need not be an integer or equal the number of coefficients. Interpretation depends on the parameterization and the chosen observational likelihood; nuisance intercepts count when they appear in that likelihood.

Suppose the Bayesian deviance is locally $D(\theta )=D(\hat{\theta })+(\theta -\hat{\theta }{)}^{T}J(\theta -\hat{\theta })$ with positive-definite $J$, and the posterior is normal with mean $m$ and covariance $\Sigma$. Writing $b=m-\hat{\theta }$, expectation and variance of the quadratic form giveConsequently the usual effective parameter count in DIC is $p_D=\mathrm{tr}(J\Sigma )$, whereas half the deviance variance is $\mathrm{tr}\{(J\Sigma {)}^2\}+2b^{T}J\Sigma Jb$. A locally flat prior gives $m=\hat{\theta }$ and $\Sigma =J^{-1}$, so both counts equal the parameter dimension. Informative priors generally destroy their equality. The expectation formula requires only the first two moments; normality is used for the variance formula.

Variable selection chooses which predictor variables enter a statistical model. The Lasso produces exact zero slopes along its Lasso regularization path; ridge regression generally shrinks without excluding variables. Selection for prediction should use predictive cross-validation or another stated criterion, rather than equating a coefficient p-value with practical usefulness.

Testing the number of components in a finite mixture model is nonregular: absent components have zero weights on the parameter boundary and their means or other parameters are unidentified. The usual chi-squared limit for a likelihood-ratio test therefore need not apply. Information criteria are useful working comparisons, while parametric bootstrap and predictive cross-validation can provide more appropriate evidence. Neither a mixture fit nor a selected component count proves genuine latent populations.

For a model with $k$ fitted parameters and maximized likelihood $L(\hat{\theta })$, the Akaike information criterion is

The corrected Akaike information criterion adds a finite-sample penalty to the Akaike information criterion, where $k$ counts fitted parameters and $n>k+1$. The displayed correction is commonly used for small samples; its exact unbiasedness depends on the model family.

Starting from a fitted model, compare admissible single-term additions and deletions within specified lower and upper scopes, retaining hierarchical lower-order terms when an interaction requires them. Select a change reducing the Akaike information criterion and repeat until no admissible change improves it. For a Gaussian normal linear model with $N$ observations and $k$ fitted mean coefficients, the criterion used by R's linear-model stepwise search is $N\log (\mathrm{RSS}/N)+2k$, up to constants common to the models. The search is greedy and need not find the best model in the whole scope. Ordinary coefficient tests in the selected model do not account for this model selection. The software arguments are documented in the MASS documentation.

Deleting $q$ fitted mean parameters improves the Akaike information criterion exactly when the displayed condition holds, since the criterion difference is $n\log ({\mathrm{RSS}}_{\mathrm{reduced}}/{\mathrm{RSS}}_{\mathrm{full}})-2q$. For one parameter, the equivalent partial F-test statistic is below $(n-p)(e^{2/n}-1)$, where $p$ is the full mean-model dimension. This is not the usual fixed-level significance-test cutoff. Fits must use the same observations and response transformation.

For a full-rank $n\times p$ normal linear model with error variance ${\sigma }^2$, the maximum-likelihood residual variance satisfiesConsequently

For a normal linear model with known error variance ${\sigma }^2$, the scaled Akaike criterion differs by a model-independent constant from

If $Y,Y^{\ast }$ are independent $N(\mu ,{\sigma }^2{I}_n)$ vectors and $H$ is a rank-$p$ orthogonal projection, thenThe first term is squared approximation bias, while $p{\sigma }^2$ is fitted-model variance and $n{\sigma }^2$ is irreducible new-response noise.

If $H$ is the rank-$p$ orthogonal projection onto a normal linear model's column space, thenAdding $2p{\sigma }^2$ gives $\Vert (I-H)\mu {\Vert }^2+(n+p){\sigma }^2$, so Mallows' $C_p$ is unbiased for independent-copy prediction error even when the projection model is misspecified.

For a model with $k$ fitted parameters, $n$ observations and maximized likelihood $L(\hat{\theta })$, the Bayesian information criterion is