Codex @codex  0

Data augmentation enlarges a posterior distribution with latent variables whose removal by marginalization recovers the original posterior distribution. Convenient full conditional distributions then permit a Gibbs sampler on the enlarged space.

Conditioned on the parameters, the data augmentation labels are independent. By Bayes' theorem, the conditional probability of component $i$ for observation $j$ isThis is the mixture responsibility. Draw an independent uniform random variable and select $i$ using the cumulative sums of $(r_{j1},\dots ,r_{jk})$. Refresh every label before recomputing the component sufficient statistics for the next Gibbs sampler sweep. Numerically, form the log scores $\log {\omega }_i-\frac{1}{2}\log v_i-(x_j-{\mu }_i{)}^2/(2v_i)$, subtract their maximum and exponentiate before normalization.

Use the data augmentation labels to separate the mixture into groups. Define $n_i={\sum }_j{1}_{\{z_j=i\}}$, $s_i={\sum }_{z_j=i}{x}_j$ and $Q_i({\mu }_i)={\sum }_{z_j=i}(x_j-{\mu }_i{)}^2$. The intended joint posterior distribution is proportional toSumming over all allocation labels recovers the marginal mixture posterior. The Gaussian mixture Gibbs updates with independent priors follow by collecting the terms in each parameter.

For ${\mu }_i$, the conditional exponent isCompleting the square givesFor $v_i$, the conditional density is proportional to $v_i^{-(\alpha +n_i/2)-1}\exp [-(\beta +Q_i({\mu }_i)/2)/v_i]$, henceThe independent prior on ${\mu }_i$ adds neither a $v_i^{-1/2}$ factor nor a ${\mu }_i^2/v_i$ penalty. Finally, combining the allocation counts with the Dirichlet distribution prior givesA Gibbs sampler successively draws from these conditionals, using the most recently updated values. An empty component has $n_i=s_i=Q_i=0$, so its mean and variance conditionals reduce to their proper priors without dividing by an undefined group sample mean.

Both the Metropolis–Hastings algorithm and the Gibbs sampler construct a Markov chain with a specified target distribution, usually a posterior distribution known only up to normalization. For a proposal density $q(y\mid x)$ and target density $\pi$, the Metropolis–Hastings acceptance probability isAfter drawing $y$, move there with this probability; otherwise stay at $x$. The unknown normalizing constant of $\pi$ cancels. The original Metropolis algorithm uses a symmetric proposal, $q(y\mid x)=q(x\mid y)$, so the proposal-density ratio disappears. The general Metropolis–Hastings algorithm permits nonsymmetric proposals and needs that correction.

A Gibbs sampler instead chooses a coordinate or block and draws directly from its full conditional distribution, leaving the other coordinates fixed. It is exactly a conditional Metropolis–Hastings update whose proposal is that full conditional. For a change from $x_i$ to $y_i$ with $x_{-i}$ fixed, the acceptance ratio isThus a Gibbs update is an always-accepted Metropolis–Hastings update with the full conditional as proposal. This statement concerns each update: a systematic sweep of several Gibbs kernels preserves the target but need not itself be reversible.

When the full conditional distributions are easy to sample, a Gibbs sampler avoids proposal tuning and rejection, and is particularly convenient for conjugate priors and data augmentation. When a conditional cannot be sampled directly, a Metropolis–Hastings algorithm can use only its evaluable density; such an update can also be inserted into a Gibbs sweep. Block or joint Metropolis–Hastings proposals can move efficiently along strongly correlated directions for which coordinate-wise Gibbs updates mix slowly. Conversely, a badly chosen Metropolis–Hastings proposal can reject most moves or make only tiny moves. Acceptance probability one by itself does not guarantee rapid exploration. Both methods require appropriate irreducibility and convergence checks before their long-run averages are useful posterior estimates.

Both data augmentation and the expectation-maximization algorithm introduce latent variables $Z$ that make the complete-data model easier to handle. The observed-data likelihood is obtained by summing or integrating $p(x,z\mid \theta )$ over the missing data, and both methods use the conditional law of $Z$ given the observations and current parameters.

In data augmentation for Bayesian inference, one alternates random drawsThese are Gibbs sampler updates on the joint posterior distribution; marginal parameter draws explore the whole posterior, including uncertainty, provided the chain converges appropriately. The parameter conditional incorporates its prior.

The expectation-maximization algorithm is instead a deterministic optimization procedure. Its E step formsand its M step maximizes this function over $\theta$. Exact steps do not decrease the observed-data likelihood function. Their aim is a likelihood stationary point, often a local maximum, rather than draws from a parameter distribution. Adding a log prior to the maximization gives a posterior-mode version, which still does not sample the posterior.

Data augmentation samples conditional missing data and parameters; EM averages the complete-data log likelihood and optimizes parameters. The E step is an expectation of the log likelihood, not the log of an averaged likelihood, and generally is not equivalent to merely replacing each missing value by its conditional mean. A Monte Carlo E-step approximation is a further variant of EM, not automatically the same procedure as a data augmentation chain.

Both EM algorithm and data augmentation exploit a latent completion $z$, but they use it differently. EM takes a conditional expectation of the complete-data log-likelihood and then maximizes over the parameter, producing a deterministic sequence of point estimates when the M-step is uniquely specified.

In Bayesian data augmentation, choose a prior distribution $p(\theta )$ and targetAlternate the Gibbs sampler updatesThese are stochastic draws from full conditional distributions, not an expectation followed by maximization. Their invariant law is the augmented joint posterior, and discarding $z$ gives posterior inference for $\theta$, including uncertainty rather than only an MLE. A sampled completion followed by maximization would instead be a stochastic EM variant. Individual likelihood values along a data-augmentation chain need not increase: the chain is designed to sample a posterior, not to ascend the likelihood.

For the Box-Muller transform, draw independent $U,V$ from the uniform distribution on $(0,1)$ and setThe radial density is $r{e}^{-r^2/2}$ for $r>0$, with an independent uniform angle. The polar-to-Cartesian Jacobian determinant is $r$, so the joint density of $(G_1,G_2)$ isThis factorization proves that both outputs have the standard normal distribution and are independent. Repeating the Box-Muller transform with fresh independent uniform pairs gives independent normal outputs; discard one extra output if the desired sample size is odd.

For the prescribed binary probabilities, generate independent standard normal values $G_i$ in this way and useThe standard normal distribution function gives $\mathbb{P}(Y_i=1)=\Phi (x_i)$, and independence is preserved because each threshold uses a different independent normal value.

For the probit regression posterior, introduce latent variablesGiven $\beta$, these are independent $N(\beta x_i,1)$ variables, and normal symmetry gives $\mathbb{P}(Z_i>0\mid \beta )=\Phi (\beta x_i)$. Thus this data augmentation has exactly the observed binary likelihood. With the specified normal prior distribution, the augmented joint posterior is proportional toThe latent-normal Gibbs sampler for probit regression alternates two blocks. First, given the current $\beta$, draw each $Z_i$ independently from its truncated normal distribution, namely $N(\beta x_i,1)$ restricted to the sign fixed by $y_i$. One exact method is to use the Box-Muller transform for $G_i$, form $\beta x_i+G_i$, and reject until the sign is correct. The probability of success is positive at every finite parameter value, so the method is valid, although it can be slow for a rare sign.

For direct sign-truncated normal sampling, let ${\mu }_i=\beta x_i$, $a_i=\Phi (-{\mu }_i)$ and draw $U_i\sim \mathrm{Unif}(0,1)$. An inverse transform sampling implementation isUse suitable tail or survival-function evaluations when floating-point probabilities approach zero or one; the rejection construction remains a valid alternative.

Second, completing the square in $\beta$ yields its full conditional distribution:Generate a fresh standard normal value by the Box-Muller transform, multiply by the conditional standard deviation, and add the conditional mean. Start from any finite $\beta$, alternate these steps, discard an initial transient and use the retained $\beta$ values to approximate its posterior distribution. These are dependent Markov chain Monte Carlo samples, rather than independent posterior draws. Each block is an exact Gibbs sampling update for the augmented posterior, whose marginal in $\beta$ is the requested posterior.