Past exam of the mathematics course of the University of Cambridge / 2003 / iii / Paper 43 / 3 / b

Use the data augmentation labels to separate the mixture into groups. Define $n_i={\sum }_j{1}_{\{z_j=i\}}$, $s_i={\sum }_{z_j=i}{x}_j$ and $Q_i({\mu }_i)={\sum }_{z_j=i}(x_j-{\mu }_i{)}^2$. The intended joint posterior distribution is proportional toSumming over all allocation labels recovers the marginal mixture posterior. The Gaussian mixture Gibbs updates with independent priors follow by collecting the terms in each parameter.

For ${\mu }_i$, the conditional exponent isCompleting the square givesFor $v_i$, the conditional density is proportional to $v_i^{-(\alpha +n_i/2)-1}\exp [-(\beta +Q_i({\mu }_i)/2)/v_i]$, henceThe independent prior on ${\mu }_i$ adds neither a $v_i^{-1/2}$ factor nor a ${\mu }_i^2/v_i$ penalty. Finally, combining the allocation counts with the Dirichlet distribution prior givesA Gibbs sampler successively draws from these conditionals, using the most recently updated values. An empty component has $n_i=s_i=Q_i=0$, so its mean and variance conditionals reduce to their proper priors without dividing by an undefined group sample mean.

Conditioned on the parameters, the data augmentation labels are independent. By Bayes' theorem, the conditional probability of component $i$ for observation $j$ isThis is the mixture responsibility. Draw an independent uniform random variable and select $i$ using the cumulative sums of $(r_{j1},\dots ,r_{jk})$. Refresh every label before recomputing the component sufficient statistics for the next Gibbs sampler sweep. Numerically, form the log scores $\log {\omega }_i-\frac{1}{2}\log v_i-(x_j-{\mu }_i{)}^2/(2v_i)$, subtract their maximum and exponentiate before normalization.