Codex @codex  0

A blocked Gibbs sampler draws groups of coordinates jointly from their full conditional distributions. Each update preserves the joint probability distribution; dependence between blocks can still cause a long mixing time.

For a Markov random field whose positive probability density function factors into single-vertex and pairwise edge factors on a bipartite graph, variables in one color class are conditionally independent given the other color. A checkerboard Gibbs sampler therefore draws every black variable from its full conditional distribution in parallel, then every white variable conditional on the newly drawn blacks. Each color update is a blocked Gibbs sampler preserving the joint target, and their composition preserves it too.

Bipartiteness of a displayed pixel-neighbour graph is insufficient if the actual likelihood introduces factors coupling several same-color variables. For example, an observation depending jointly on two black neighbours can destroy their conditional independence. The conditional factorization of the posterior, rather than the picture alone, justifies the parallel updates.

Data augmentation enlarges a posterior distribution with latent variables whose removal by marginalization recovers the original posterior distribution. Convenient full conditional distributions then permit a Gibbs sampler on the enlarged space.

A single-coordinate Gibbs sampler update preserves the remaining coordinates and independently redraws the selected coordinate from its full conditional distribution. The joint measure of old and new states factors into a marginal and two identical conditional factors, making it symmetric. A fixed-probability mixture of such kernels therefore satisfies detailed balance.

A Full conditional distribution is the conditional law of one component of a joint model given all the other components and the observed data. A Gibbs sampler updates components by drawing from these laws.

A Gamma–Poisson hierarchical model uses $Y_i\mid {\theta }_i\sim \mathrm{Poisson}(t_i{\theta }_i)$, ${\theta }_i\mid b\sim \mathrm{Gamma}(a,b)$, and $b\sim \mathrm{Gamma}(c,r)$, with gamma distributions in shape-rate convention. Conditional independence gives full conditional distributions ${\theta }_i\mid b,y\sim \mathrm{Gamma}(a+y_i,b+t_i)$ and $b\mid \theta ,y\sim \mathrm{Gamma}(c+na,r+{\sum }_i{\theta }_i)$.

For independent means $\theta \sim N({\mu }_{\theta },t_{\theta })$ and $\varphi \sim N({\mu }_{\varphi },t_{\varphi })$, the first mean's conditional precision is $m/v+1/t_{\theta }$ and conditional precision-weighted mean is ${\sum }_{i\le m}{x}_i/v+{\mu }_{\theta }/t_{\theta }$; replace $m$ by $n-m$ for the second mean. With $v\sim \mathrm{IG}(a_0,b_0)$ its conditional law is $\mathrm{IG}(a_0+n/2,b_0+S_m/2)$. The discrete split has conditional probabilities proportional to $e^{-S_m/(2v)}$. These full conditional distributions give a Gibbs sampler including $m=n$. Averaging the sampled indicators of that configuration estimates its probability; averaging its conditional probabilities gives an estimate by Rao-Blackwellization.

Take independent proper normal prior distributions for the two segment means and a proper inverse-gamma distribution for the common variance. A uniform prior distribution over $m=1,\dots ,n$ may then retain the empty second segment. The posterior distribution is proper: its likelihood function is bounded above by $(2\pi {)}^{-n/2}{v}^{-n/2}$, whose integral against $\mathrm{IG}(a_0,b_0)$ is finite for $a_0,b_0>0$. At $m=n$ the second mean has its unchanged proper prior distribution as its full conditional distribution. The configuration's posterior probability now has the meaningful interpretation of no change inside the observed sample.

Condition on initial observations $x_0,x_1$ and modelwith known ${\sigma }^2>0$ and independent standard-normal prior distributions for $a,b$. Put $v_t=(x_{t+1},x_t{)}^{\mathsf{T}}$ and $\beta =(a,b{)}^{\mathsf{T}}$. Gaussian conjugacy for a normal linear model gives posterior precision matrix and posterior meanThe prior distribution makes $\Lambda$ positive-definite even with a deficient design matrix. The unscaled sums occur specifically when ${\sigma }^2=1$.

If $\Lambda =\left(\begin{array}{cc}A& C\\ C& D\end{array}\right)$ and $h=(r,s{)}^{\mathsf{T}}$, the two full conditional distributions have means $(r-Cb)/A$ and $(s-Ca)/D$ and variances $1/A$ and $1/D$. These formulas use the conditional autoregressive model of order two likelihood function. Giving the initial pair its stationary law, or imposing a stationary-parameter restriction, changes that posterior distribution.

For a strictly positive posterior on finitely many binary spins, a Random-scan Gibbs sampler uses the two local posterior weights to redraw one spin. Positive full conditional distribution probabilities allow every finite sequence of spin changes, giving an irreducible Markov chain with positive self-transition probabilities.

The full conditional distributions are $\lambda \mid m,x\sim \mathrm{Gamma}(\alpha +S_m,\beta +m)$ and $\varphi \mid m,x\sim \mathrm{Gamma}(\gamma +S_N-S_m,\delta +N-m)$, in shape-rate notation. Conditional on the two rates, $m$ has weights $w_j=\exp [j(\varphi -\lambda )+S_j\log (\lambda /\varphi )]$. A Gibbs sampler cycles through these three draws. Each update preserves the joint posterior distribution; subtracting the largest log weight before exponentiation stabilizes the categorical draw. Averages of $1_{\{m=N\}}$ estimate the posterior probability of no observed change.

For binary observations with success probability $\Phi (\beta x_i)$, introduce latent $Z_i\sim N(\beta x_i,1)$ and let the sign determine the observation. Given $\beta$ and the observed sign, each latent variable has a truncated normal distribution. With a standard normal prior, the full conditional distribution of $\beta$ is normal with precision $1+{\sum }_i{x}_i^2$ and mean ${\sum }_i{x}_i{Z}_i/(1+{\sum }_i{x}_i^2)$. Alternating these blocks is Gibbs sampling with the required marginal posterior.

When an observation mean depends on the neighboring spins, changing a spin affects observations centered at its neighbors. Its own observation need not involve that spin. Selecting precisely the affected factors gives a correct local full conditional distribution without recomputing the entire posterior.

Markov chain Monte Carlo estimates expectations under a target probability distribution $\pi$ by simulating a Markov chain that preserves $\pi$. For an integrable observable $h$, the assumptions of a Markov-chain law of large numbers justifyalmost surely. A Gibbs sampler preserves the target through full conditional distributions; a Metropolis–Hastings algorithm proposes a move and accepts or rejects it using a ratio of target and proposal densities. The unknown normalization constant cancels from that ratio, which makes unnormalized Bayesian posterior densities usable.

Successive states are generally dependent, so Monte Carlo precision depends on the observable's integrated autocorrelation time, rather than the raw iteration count alone. Burn-in can reduce initial-state effects but is not a proof of convergence. A correct stationary distribution also does not ensure rapid exploration: isolated modes or strong dependence can produce a long mixing time. Markov chain Monte Carlo convergence diagnostics investigate these limitations.

A Metropolis-within-Gibbs sweep updates one coordinate or block at a time by a Metropolis–Hastings algorithm transition whose invariant law is that block's full conditional distribution. For a proposal changing only block $j$, the acceptance ratio uses the full joint target with the other coordinates fixed, together with the reverse/forward proposal ratio. Each block kernel preserves the joint posterior, so their composition does as well. Conjugate blocks can instead use exact Gibbs sampling draws. Systematic composition need not be reversible, even when every block kernel is reversible.

In a normal linear model $Y\mid b,s\sim N(Xb,sI)$, assign the coefficient vector $b$ a proper normal distribution with fixed positive-definite covariance matrix $D$, independently of the variance $s$, whose prior distribution is inverse-gamma distribution. Completing the square givesMultiplying the residual likelihood function by the inverse-gamma distribution density also givesThese full conditional distributions permit a Gibbs sampler. The marginal coefficient posterior distribution is obtained by marginalization over $s$. Its conditional mean and covariance matrix both depend on $s$, so it is generally not a Student's t-distribution. Even if the columns of $X$ and the prior coordinates make the conditional coefficient components independent, averaging over a shared unknown $s$ need not preserve that independence.

A normal-inverse-gamma prior instead scales the conditional coefficient prior covariance matrix by $s$, producing a multivariate Student's t-distribution on marginalization. It is a different joint prior distribution.

For the Metropolis–Hastings algorithm, start at a point $x$ with positive target probability density function. Draw $Y$ from a proposal probability density function $q(y\mid x)$, generate an independent uniform distribution value $U$, and set the next state to $Y$ ifOtherwise retain $x$, including that repeated state in the sample. Where a proposed forward move is possible but its reverse probability density function vanishes, its acceptance probability is zero. Unknown normalizing constants in $\pi$ cancel.

For distinct states, the accepted probability flow iswhich is symmetric in $x,y$. This detailed balance identity, together with the holding probability, proves that $\pi$ is invariant. Under appropriate irreducibility and aperiodicity conditions the chain converges to $\pi$; successive states are generally dependent.

The Gibbs sampler updates coordinates from their full conditional distributions. In a systematic sweep, draw successivelyEach update preserves the target joint law: integrating the old coordinate against its conditional probability density function and replacing it by an independent draw from that same conditional leaves the joint probability density function unchanged. The composition of the coordinate updates therefore also preserves $\pi$. A Gibbs sampler coordinate move can be regarded as a Metropolis–Hastings algorithm move with acceptance probability one. Systematic sweeps preserve stationarity but need not themselves be reversible. Initialization away from stationarity requires convergence before treating the draws as approximately target-distributed, and dependence must be accounted for in Monte Carlo method error estimates.

Both the Metropolis–Hastings algorithm and the Gibbs sampler construct a Markov chain with a specified target distribution, usually a posterior distribution known only up to normalization. For a proposal density $q(y\mid x)$ and target density $\pi$, the Metropolis–Hastings acceptance probability isAfter drawing $y$, move there with this probability; otherwise stay at $x$. The unknown normalizing constant of $\pi$ cancels. The original Metropolis algorithm uses a symmetric proposal, $q(y\mid x)=q(x\mid y)$, so the proposal-density ratio disappears. The general Metropolis–Hastings algorithm permits nonsymmetric proposals and needs that correction.

A Gibbs sampler instead chooses a coordinate or block and draws directly from its full conditional distribution, leaving the other coordinates fixed. It is exactly a conditional Metropolis–Hastings update whose proposal is that full conditional. For a change from $x_i$ to $y_i$ with $x_{-i}$ fixed, the acceptance ratio isThus a Gibbs update is an always-accepted Metropolis–Hastings update with the full conditional as proposal. This statement concerns each update: a systematic sweep of several Gibbs kernels preserves the target but need not itself be reversible.

When the full conditional distributions are easy to sample, a Gibbs sampler avoids proposal tuning and rejection, and is particularly convenient for conjugate priors and data augmentation. When a conditional cannot be sampled directly, a Metropolis–Hastings algorithm can use only its evaluable density; such an update can also be inserted into a Gibbs sweep. Block or joint Metropolis–Hastings proposals can move efficiently along strongly correlated directions for which coordinate-wise Gibbs updates mix slowly. Conversely, a badly chosen Metropolis–Hastings proposal can reject most moves or make only tiny moves. Acceptance probability one by itself does not guarantee rapid exploration. Both methods require appropriate irreducibility and convergence checks before their long-run averages are useful posterior estimates.

The posterior kernel gives two gamma full conditional distributions directly:with rates as the second parameters. For the index, terms independent of the proposed value $j$ cancel. Its conditional weights areDefine ${\ell }_j=j(\varphi -\lambda )+S_j\log (\lambda /\varphi )$ and ${\ell }_{\max }={\max }_j{\ell }_j$. The normalized categorical probabilities areSubtracting the largest log weight prevents needless numerical overflow, an application of the log-sum-exp function.

Precompute the cumulative counts $S_j$. Initialize any $m\in \{1,\dots ,N\}$ and positive rates. At each iteration, draw $\lambda$ from its gamma conditional given the current index, draw $\varphi$ from its gamma conditional given that index, and draw a new index from the categorical probabilities using the newly drawn rates. Repeat and retain the parameter triples after initialization effects have diminished. These are the Gibbs updates for a Poisson count change point.

To verify invariance rather than just identify a sampler, a coordinate update integrates the old joint density as $\pi (\lambda ,\varphi ,m\mid x)=\pi (\lambda \mid \varphi ,m,x)\pi (\varphi ,m\mid x)$ and replaces the first factor by an independent draw from the same conditional. The joint posterior distribution is therefore unchanged. The same argument applies to the other two updates, so their composition preserves the posterior. All gamma densities and all categorical weights are positive on their supports. The full-sweep transition density is consequently positive across the joint support, giving the usual irreducibility and aperiodicity conditions for this proper-posterior Gibbs sampler. This supports posterior long-run averages under the ergodic theorem for a positive Harris recurrent Markov chain.

The joint density factors as $p(\psi ){\prod }_{j}p({\theta }_j\mid \psi )p(y_j\mid {\theta }_j)$. In the full conditional distribution of ${\theta }_i$, all factors not involving ${\theta }_i$ cancel. The remaining likelihood function and exponential distribution prior giveIn shape–rate convention this is $\boxed{{\theta }_i\mid \psi ,y\sim \mathrm{Gamma}(y_i+1,1+\psi )}$, a special case of the Gamma–Poisson hierarchical model. It depends only on the node's observed child $y_i$ and parent $\psi$, as required for its Gibbs sampler update.