Past exam of the mathematics course of the University of Cambridge / 2010 / iii / Paper 36 / 3

The logarithm is strictly increasing, so maximizing the likelihood function is equivalent to maximizing its logarithm. Multiplication by $-2$ reverses that ordering. Thus, whenever the maximum is attained,Here the Bayesian deviance includes the likelihood constants. Parameter-independent constants do not alter the minimizer, but a common convention must be retained when comparing criteria between models.

Under the usual regular, identifiable, fixed-dimensional setting, the maximum-likelihood estimator is an interior point with vanishing log-likelihood gradient, and the observed information is positive definite. DefineThe second-order Taylor expansion of the Bayesian deviance isWith a prior density approximately constant over the likelihood's concentration region, the posterior density is therefore proportional toRecognizing the multivariate normal density givesTypically $J$ grows at rate $n$ and the posterior concentrates in a region of radius $n^{-1/2}$, making higher-order terms negligible under the regularity assumptions. The local flatness assumption concerns this region, not necessarily the entire parameter space. Boundary maxima, unidentified directions, or persistent multiple modes can invalidate this approximation.

Conditional on the observed data, the preceding multivariate normal distribution approximation implies thatThe deviance expansion consequently givesThe sum of squares of $p$ independent standard normal distribution variables has chi-squared distribution with $p$ degrees of freedom. Hence, as a statement about the posterior distribution of the Bayesian deviance,The data are fixed here and the parameter varies according to its posterior; this is not a claim about resampling the data.

Let $\overline{\theta }=\mathbb{E}(\theta \mid y)$ and $\overline{D}=\mathbb{E}\{D(\theta )\mid y\}$. The usual effective parameter count in DIC isWith the locally flat prior distribution, $\overline{\theta }\approx \hat{\theta }$ and the mean of the chi-squared distribution excess is $p$. Thus $\overline{D}\approx D(\hat{\theta })+p$, while $D(\overline{\theta })\approx D(\hat{\theta })$, giving $p_D\approx p$.

More generally, the quadratic posterior deviance moments give the effective count. Within the same approximation, let the posterior covariance matrix be $\Sigma$. The expectation of a quadratic form giveswhereas evaluation at $\overline{\theta }$ omits the trace term. Therefore $p_D\approx \mathrm{tr}(J\Sigma )$. For a flat prior, $\Sigma \approx J^{-1}$ and the trace is $p$; informative prior shrinkage can reduce this effective count.

Using the same Bayesian deviance convention, the deviance information criterion and Akaike information criterion areBoth combine a lack-of-fit term with a complexity penalty; smaller values are preferred among comparable models. Under the locally flat prior distribution and the regular multivariate normal distribution approximation, $D(\overline{\theta })\approx D(\hat{\theta })$ and $p_D\approx p$, soAll fitted parameters counted in the likelihood, including unknown dispersion parameters, belong in $p$. The equivalence need not hold for informative priors, nonregular models, or different choices of the likelihood's latent-variable representation.

For retained Markov chain Monte Carlo draws ${\theta }^{(1)},\dots ,{\theta }^{(S)}$, calculate both the parameter average $\hat{m}=S^{-1}{\sum }_s{\theta }^{(s)}$ and the average deviance ${\hat{D}}_{\mathrm{av}}=S^{-1}{\sum }_{s}D({\theta }^{(s)})$. Two estimates areThe first uses the approximate equality of the posterior mean and maximum-likelihood estimator. The second subtracts the mean $p$ of the chi-squared distribution deviance excess. These are different operations: evaluation after averaging versus averaging after evaluation. Both rely on a well-mixed chain and the regular locally flat-prior approximation. The smallest sampled deviance is an upper bound on the true minimum, but merely taking that minimum can be inefficient in high dimension because few draws approach the mode closely.

The alternative effective count isUnder the locally flat-prior approximation, the constant $D(\hat{\theta })$ contributes no variance, and the remaining chi-squared distribution term has variance $2p$. Hence $p_V\approx p$, the same calibration as the usual effective parameter count in DIC. It can be estimated by half the sample variance of deviance draws, without evaluating deviance at a parameter average.

This is a calibrated alternative, not a general identity with $p_D$. For example, with a quadratic deviance and a multivariate normal distribution posterior of mean $m$ and covariance $\Sigma$, putting $b=m-\hat{\theta }$ givesThey agree when $b=0$ and $\Sigma =J^{-1}$, but informative priors can change both the covariance and the displacement of the posterior mean. Their equality should therefore not be assumed outside the stated regime.