Past exam of the mathematics course of the University of Cambridge / 2006 / iii / Paper 47 / 5

For the Metropolis–Hastings algorithm, start at a state $\theta$ with positive target density, propose ${\theta }^{\prime }$ from $q({\theta }^{\prime }\mid \theta )$, and generate an independent uniform draw. Accept the proposal with probabilityOn rejection retain $\theta$, including repeated states in the chain; removing repeats would generally change its stationary law. The accepted-flow density satisfieswhich is symmetric in its two states. This proves detailed balance and hence invariance of the target distribution, including the holding probabilities on the diagonal. Unknown common normalization constants cancel in the ratio. Invariance alone does not establish convergence from arbitrary initial states. Choosing proposals that yield a positive Harris recurrent Markov chain with aperiodicity supplies the usual convergence guarantee, and the resulting sample is dependent. Use burn-in and account for autocorrelation when assessing Monte Carlo precision.

A random-walk update proposes ${\theta }^{\prime }=\theta +\eta$, with an increment distribution independent of the current state. If its density is symmetric, $q({\theta }^{\prime }\mid \theta )=q(\theta \mid {\theta }^{\prime })$, reducing the Metropolis–Hastings acceptance probability to $\min \{1,\pi ({\theta }^{\prime })/\pi (\theta )\}$. For nonsymmetric increments the proposal-density ratio must remain.

Such updates need little global knowledge of the target and adapt naturally to its local scale. Their limitation is a tuning tradeoff: very small increments accept often but move slowly, while very large increments are often rejected. They can also have difficulty moving between distant modes. Covariance-scaled increments help with anisotropic targets, but the chain still explores through local steps.

An independence sampler proposes from a fixed density $g({\theta }^{\prime })$, independent of the current state. Its acceptance probability isIf $g$ is close to the target, proposals can make large efficient moves and cross separated modes; $g=\pi$ gives acceptance one and independent samples. On the other hand, a good global approximation is required. A proposal with tails too light can leave the chain trapped at states having very large importance weight $\pi /g$, since most outgoing proposals then have tiny acceptance probabilities.

For perspective, if the normalized target satisfies $\pi \le Mg$, the accepted transition density is at least $\pi ({\theta }^{\prime })/M$, because both entries in $\min \{g({\theta }^{\prime }),\pi ({\theta }^{\prime })/w(\theta )\}$ have that lower bound, where $w=\pi /g\le M$. This yields a uniform refresh component. Thus a well-designed independence proposal can mix rapidly, whereas a poor global proposal can be much worse than a locally tuned random walk. The tradeoff concerns effective samples per computation, not acceptance rate alone.

Let $v={\sigma }^2>0$, $d_k=k+1$, and $S_k(a)=\Vert y-X_{k}a{\Vert }^2$. Under the standard interpretation that the two stated priors are independent, multiply the normal likelihood by the normal coefficient prior and the inverse-gamma distribution density. The independent normal and inverse-gamma regression priors giveFor a fixed model the determinant and Gaussian normalizing constants are independent of $a_k,v$ and may be omitted here; ${\Sigma }_k$ is assumed positive-definite. The coefficient prior is not scaled by $v$, so replacing its precision by ${\Sigma }_k^{-1}/v$ would describe a different prior.

For an additional check, completing the coefficient square and reading the variance kernel give the full conditional distributionsandMarginal priors alone would not determine their joint prior without the independence assumption. The front-page inverse-gamma mean has a typographical error: this density has mean $\beta /(\alpha -1)$ when $\alpha >1$, not a denominator $\alpha +1$. The density itself, used in the posterior calculation, fixes the correct parameterization.

The cross-model target must include a model-order prior ${\rho }_k$. Write its joint, unnormalized density asHere the prior densities are normalized within each model. In particular their $k$-dependent determinants and powers of $2\pi$ cannot be dropped merely because the fixed-model posterior in part (i) was given up to proportionality.

For explicit birth and death moves for Bayesian variable selection, choose a birth with probability $b_k$, generate $u$ from a density $g_k(u\mid a,v)$, append $u$ as the new covariate coefficient, and leave $v$ unchanged. The map from $(a,v,u)$ to $(a^{\prime },v^{\prime })=((a,u),v)$ is invertible under the reverse deletion and has absolute Jacobian one. It matches dimensions: the smaller-model coefficients and variance have dimension $k+2$, and the one auxiliary draw supplies the larger-model dimension $k+3$. If the reverse death is selected with probability $d_{k+1}$, accept withFor a death, delete the last coefficient $u$, retain the others and the variance, and useTo verify detailed balance, multiply the birth acceptance by $t_k{b}_k{g}_k$ and the reverse death acceptance by $t_{k+1}{d}_{k+1}$. Both equal the minimum of these two quantities in the matched coordinates. This proves reversibility without merely naming a dimension-changing algorithm. A more general bijection introduces its absolute Jacobian, and random choices among several covariates introduce the corresponding forward and reverse selection probabilities. Add within-model updates so that coefficients and variance can explore each model. These are reversible-jump Markov chain Monte Carlo updates; the unspecified model prior and proposal density must be supplied to implement them numerically.