A Gibbs sampler constructs a Markov chain by repeatedly drawing coordinates from their full conditional distributions given the current values of the other coordinates. For a two-coordinate target density , a sweep draws and then . Each update preserves the target joint law: retaining one marginal and resampling its exact conditional reconstructs the same joint density. Thus a sweep has the target as a stationary distribution. Convergence from arbitrary starts additionally needs suitable irreducibility and other conditions; disconnected support can trap a chain. Successive output samples are usually dependent.
In a systematic Gibbs sampler for a standardized bivariate normal distribution with correlation , update conditionally on the old , then update using the new . Substitution gives the displayed autoregressive process of order one, with independent noises with the standard normal distribution . The sweep-to-sweep autocorrelation of at lag is , so the chain mixes slowly as approaches one.
For a standard bivariate normal distribution with correlation , each conditional has variance and mean times the other coordinate. A sweep draws , then . The resulting coordinate chain is an AR(1) process with coefficient and innovation variance . It shows explicitly why near-unit correlation slows successive Gibbs sweeps.
Software for Bayesian inference through Markov chain Monte Carlo in directed statistical models. Its model language uses
dnorm(mean, precision), so the second argument is inverse variance; dcat indexes a supplied probability vector, and dbern introduces a binary latent variable. The model is declarative, so the order of deterministic node definitions does not impose procedural evaluation order.For Gaussian precision , a systematic update of the first coordinate followed by the second makes the centered second coordinate an AR(1) chain with coefficient . Independent conditional Gaussian noise supplies its stable innovation. This proves convergence of the Gaussian Gibbs sampler and explains slower mixing when posterior correlation has magnitude near one.
Every full-conditional coordinate update satisfies detailed balance with the target distribution. Their systematic ordered composition preserves the target but need not satisfy detailed balance, since compositions of reversible kernels need not be reversible. A fixed-probability random-scan mixture remains reversible.
A blocked Gibbs sampler draws groups of coordinates jointly from their full conditional distributions. Each update preserves the joint probability distribution; dependence between blocks can still cause a long mixing time.
For a Markov random field whose positive probability density function factors into single-vertex and pairwise edge factors on a bipartite graph, variables in one color class are conditionally independent given the other color. A checkerboard Gibbs sampler therefore draws every black variable from its full conditional distribution in parallel, then every white variable conditional on the newly drawn blacks. Each color update is a blocked Gibbs sampler preserving the joint target, and their composition preserves it too.
Bipartiteness of a displayed pixel-neighbour graph is insufficient if the actual likelihood introduces factors coupling several same-color variables. For example, an observation depending jointly on two black neighbours can destroy their conditional independence. The conditional factorization of the posterior, rather than the picture alone, justifies the parallel updates.
A random-scan Gibbs sampler chooses one coordinate, commonly uniformly at random, and redraws it from its complete conditional distribution while leaving all other coordinates fixed. Each coordinate update preserves the target distribution.
A single-coordinate Gibbs sampler update preserves the remaining coordinates and independently redraws the selected coordinate from its full conditional distribution. The joint measure of old and new states factors into a marginal and two identical conditional factors, making it symmetric. A fixed-probability mixture of such kernels therefore satisfies detailed balance.
Tempered Gibbs sampling combines modified full conditional distributions with state-dependent coordinate selection and importance sampling. For a positive target density , choose strictly positive normalized densities with compatible support, and defineSelect coordinate proportionally to and redraw it from . The invariant density is , not generally . For a move changing only coordinate , stationary transition weights are proportional to , symmetric in old and new values. Integrating also shows .
Under the appropriate Markov-chain law of large numbers, weighted averages with weights recover expectations under . Flattening a conditional can encourage longer moves, but omitting the weights changes the estimand. The need not be full conditionals of a common second joint distribution.[ref]
For independent , a flat prior on , and an exponential prior of rate on , the full conditionals areandwhere the gamma distribution is parametrized by shape and rate.
Articles by others on the same topic
There are currently no matching articles.