Past exam of the mathematics course of the University of Cambridge / 2003 / iii / Paper 43 / 3 / a

Write $v_i={\sigma }_i^2$ for each component variance and ${\tau }^2$ for the fixed prior variance denoted ${\sigma }^2$ in the source; these must not be confused. Let $\phi (x;\mu ,v)$ be the normal distribution probability density function and $D_j={\sum }_{i=1}^k{\omega }_i\phi (x_j;{\mu }_i,v_i)$. Independent observations and independent parameter priors give the posterior distributionIts support has $v_i>0$, ${\omega }_i>0$ and ${\sum }_i{\omega }_i=1$, with $\alpha ,\beta ,{\tau }^2,{ϵ}_i>0$. The component means have independent fixed-variance priors with a normal distribution, not means with variances proportional to $v_i$. The factor $e^{-\beta /v_i}$ follows from the stated inverse-gamma distribution. The later printed augmented expression instead has $e^{-\beta v_i}$ and an undefined prior mean; those are inconsistent with this specification. The intended model uses $e^{-\beta /v_i}$ and prior mean zero throughout.

Hold all parameters except ${\mu }_i$ fixed and propose ${\mu }_i^{\prime }={\mu }_i+\eta$ with $\eta \sim N(0,s_{\mu i}^2)$. This random-walk Metropolis algorithm has a symmetric proposal distribution. DefineCancellation of unchanged likelihood and prior factors gives the Metropolis–Hastings acceptance probabilityDraw an independent uniform and accept if it is at most $a_i$; otherwise retain the old value. Update each component in turn, recomputing the needed mixture contributions. On a logarithmic scale, compare $\log U$ with $\min (0,{\sum }_j\log (D_j^{\prime }/D_j)-[({\mu }_i^{\prime }{)}^2-{\mu }_i^2]/(2{\tau }^2))$ to avoid numerical underflow.

A valid symmetric proposal distribution is $v_i^{\prime }=v_i+\eta$ with $\eta \sim N(0,s_{vi}^2)$ on the entire real line. Reject immediately if $v_i^{\prime }\le 0$. Otherwise use the positive-parameter random walk with boundary rejection. Withthe Metropolis–Hastings acceptance probability isNo Jacobian determinant is needed for this additive proposal in $v_i$. Do not repeatedly redraw until $v_i^{\prime }>0$: that would create a state-dependent truncated proposal and require an additional Hastings correction. Alternatively a symmetric walk in $u_i=\log v_i$ respects positivity, but then the transformed target has a factor $v_i$; equivalently the acceptance ratio in variance coordinates has the extra factor $v_i^{\prime }/v_i$.

Choose a fixed Dirichlet distribution proposal $q(\mathbf{\omega })\propto {\prod }_i{\omega }_i^{{\eta }_i-1}$ with every ${\eta }_i>0$, independent of the current state, and propose the entire weight vector. This automatically respects the simplex constraint. With $D_j^{\prime }={\sum }_i{\omega }_i^{\prime }\phi (x_j;{\mu }_i,v_i)$, the Independence Metropolis–Hastings algorithm accepts withThis includes $q(\mathbf{\omega })/q({\mathbf{\omega }}^{\prime })$, not merely the posterior ratio. A particularly simple choice is ${\eta }_i={ϵ}_i$, the weight prior: its factors cancel and the ratio is just the mixture likelihood ratio. A proposal concentrated around a pilot estimate can mix better, provided it retains positive density throughout the relevant simplex. Both target and proposal densities use the same $k-1$ independent simplex coordinates, so this direct weight proposal has no extra transformation Jacobian determinant.