Gibbs sampler

For a joint density $f_{XY}$, a two-coordinate Gibbs update draws

In a systematic Gibbs sampler for a standardized bivariate normal distribution with correlation $\rho$, update $Z_1$ conditionally on the old $Z_2$, then update $Z_2$ using the new $Z_1$. Substitution gives the displayed autoregressive process of order one, with independent noises with the standard normal distribution ${\eta }_{1,r},{\eta }_{2,r}$. The sweep-to-sweep autocorrelation of $Z_2$ at lag $h$ is ${\rho }^{2h}$, so the chain mixes slowly as $|\rho |$ approaches one.

For a standard bivariate normal distribution with correlation $\rho$, each conditional has variance $1-{\rho }^2$ and mean $\rho$ times the other coordinate. A sweep draws $X_{n+1}=\rho Y_n+\sqrt{1-{\rho }^2}{Z}_1$, then $Y_{n+1}=\rho X_{n+1}+\sqrt{1-{\rho }^2}{Z}_2$. The resulting coordinate chain is an AR(1) process with coefficient ${\rho }^2$ and innovation variance $1-{\rho }^4$. It shows explicitly why near-unit correlation slows successive Gibbs sweeps.

Software for Bayesian inference through Markov chain Monte Carlo in directed statistical models. Its model language uses dnorm(mean, precision), so the second argument is inverse variance; dcat indexes a supplied probability vector, and dbern introduces a binary latent variable. The model is declarative, so the order of deterministic node definitions does not impose procedural evaluation order.

For Gaussian precision $\left(\begin{array}{cc}A& C\\ C& D\end{array}\right)$, a systematic update of the first coordinate followed by the second makes the centered second coordinate an AR(1) chain with coefficient $C^2/(AD)<1$. Independent conditional Gaussian noise supplies its stable innovation. This proves convergence of the Gaussian Gibbs sampler and explains slower mixing when posterior correlation has magnitude near one.

For a strictly positive posterior on finitely many binary spins, a Random-scan Gibbs sampler uses the two local posterior weights to redraw one spin. Positive full conditional distribution probabilities allow every finite sequence of spin changes, giving an irreducible Markov chain with positive self-transition probabilities.

When an observation mean depends on the neighboring spins, changing a spin affects observations centered at its neighbors. Its own observation need not involve that spin. Selecting precisely the affected factors gives a correct local full conditional distribution without recomputing the entire posterior.

Every full-conditional coordinate update satisfies detailed balance with the target distribution. Their systematic ordered composition preserves the target but need not satisfy detailed balance, since compositions of reversible kernels need not be reversible. A fixed-probability random-scan mixture remains reversible.

A blocked Gibbs sampler draws groups of coordinates jointly from their full conditional distributions. Each update preserves the joint probability distribution; dependence between blocks can still cause a long mixing time.

On a bipartite image graph, the pixels of one color are conditionally independent given the other color. Draw all black pixels from their independent full conditionals, then draw all whites conditional on the new blacks. Each color update preserves the target law, so their composition does too.

A random-scan Gibbs sampler chooses one coordinate, commonly uniformly at random, and redraws it from its complete conditional distribution while leaving all other coordinates fixed. Each coordinate update preserves the target distribution.

A single-coordinate Gibbs sampler update preserves the remaining coordinates and independently redraws the selected coordinate from its full conditional distribution. The joint measure of old and new states factors into a marginal and two identical conditional factors, making it symmetric. A fixed-probability mixture of such kernels therefore satisfies detailed balance.

A tempered Gibbs sampler changes the coordinate-selection probabilities and compensates by drawing from modified complete conditionals. Its invariant law can differ from the desired target, so expectations under the target are recovered with explicit importance weights.

The joint target density is stationary for a full Gibbs sweep, since

For independent $Z_i\mid \mu ,\omega \sim N(\mu ,{\omega }^{-1})$, a flat prior on $\mu$, and an exponential prior of rate $\lambda$ on $\omega$, the full conditionals areandwhere the gamma distribution is parametrized by shape and rate.