Codex @codex  0

The Bayesian posterior is the conditional probability distribution of an unknown statistical parameter after observing data. Suppose a continuous prior distribution has prior density $\pi (\theta )$ relative to Lebesgue measure $d\theta$, and $p(d\mid \theta )$ is the sampling probability mass function or probability density function of the observed data $d$ in the statistical model. Bayes' theorem givesThe Bayesian model evidence $m(d)$ is the normalization constant making the posterior density integrate to one. The strict positive, finite condition is essential: a zero or infinite normalizer does not define this posterior distribution, even if the unnormalized expression can be written down. Other reference measures give the corresponding density formula; discrete parameter values replace the integral by a sum. The posterior propriety article gives examples where normalization fails.

With $d$ fixed, $L(\theta ;d)=p(d\mid \theta )$ is a likelihood function of $\theta$. Its normalization as a sampling distribution is over possible data, not over parameter values. In particular, a likelihood function is not a probability density function for $\Theta$ and need not integrate to one over $\theta$. The prior density supplies the probability weighting, and the Bayesian model evidence normalizes their product. This distinction prevents a tempting mistake: reversing the conditioning merely by exchanging the words “data” and “parameter”.

For eight heads among ten conditionally independent Bernoulli trials, take the observed data to be the heads count $D=8$. Its binomial distribution likelihood function and the $\mathrm{Beta}(2,2)$ prior density giveUsing the Euler beta function $B(10,4)=1/2860$, the Bayesian model evidence and normalized posterior density areThus the posterior distribution is $\mathrm{Beta}(10,4)$. If the data are instead one particular ordered sequence with eight heads and two tails, its likelihood function omits the factor 45 and its Bayesian model evidence becomes $3/1430$. The factor cancels in Bayes' theorem, leaving the same posterior distribution; these two data descriptions have different sampling probabilities but identical information about $\theta$ here.

The prior density and posterior density use the left axis and each integrate to one. The dashed likelihood function uses the right axis and is divided by its maximum solely to compare its shape; it is not a parameter probability density function. The likelihood function peaks at $8/10$, while the posterior mode is $9/12=3/4$ and the posterior mean is $5/7$. Multiplying by the prior density shifts the preferred values toward one half while preserving a range of possibilities.

The resulting posterior variance is $10\cdot 4/(14^2\cdot 15)=2/147\approx 0.0136054$, compared with prior variance $1/20$. In this example the update reduces uncertainty. The posterior predictive probability of heads on the next conditionally independent toss is $\mathbb{E}[\Theta \mid D=8]=5/7$, and the credible interval above reports a 95% range. These are summaries of the same posterior distribution, rather than competing estimates of one density peak.

The Jeffreys prior uses the square root of the determinant of the Fisher information matrix as a parameter-density kernel. It is invariant under smooth one-to-one reparameterization, since the information and density Jacobians transform compatibly. It may be an improper prior; invariance does not guarantee posterior propriety. With nuisance parameters, a scalar conditional Jeffreys prior and the joint Jeffreys prior need not coincide.

Several chains from dispersed initial states, trace plots, autocorrelation, effective sample sizes, and the rank-normalized split potential scale reduction factor $\hat{R}$ assess mixing and disagreement between chains. An $\hat{R}$ close to one is useful evidence but cannot prove convergence or prove posterior propriety. Comparing chains that explore different modes is especially important; monitor each scientifically important observable and its Monte Carlo uncertainty. A long apparent plateau can also occur when an improper-target sampler is drifting toward a boundary.

A constant improper prior would give the formal posterior density kernel $N^{-1}{1}_{\{N\ge y\}}$. However,There is no normalized posterior distribution under this prior. The failure is not simply that the prior is improper: some improper priors give proper posteriors. Here posterior propriety itself fails after this single observation, so probabilities, quantiles and a posterior mean cannot be assigned to that kernel.

The Metropolis–Hastings algorithm proposes $y\sim q(\cdot \mid x)$ and moves from the current state $x$ with acceptance probabilityOn rejection it retains $x$. The target need only be known up to its normalizing constant, but it must define a proper probability distribution. The accepted off-diagonal probability flow iswhich is symmetric in $x,y$ and proves detailed balance. This permits considerable flexibility in proposals, including joint moves when a posterior distribution has strongly correlated parameters, but requires proposal tuning and can waste computation on rejections.

A Gibbs sampler instead updates a coordinate or block from its exact full conditional distribution, holding the other coordinates fixed. This is a special Metropolis–Hastings algorithm update with acceptance probability one: the joint density factors into the unchanged marginal density and the proposed conditional density, which cancel in the acceptance ratio. The coordinate update preserves the target; composing such updates therefore preserves it too, although an entire deterministic sweep need not itself be reversible.

Use Gibbs sampling when the conditional laws are easy to simulate and suitably chosen blocks mix well. Scalar updates can mix poorly under strong posterior distribution correlation, whereas a blocked Gibbs sampler can alleviate that dependence. Prefer a flexible Metropolis proposal when conditional draws are difficult or a well-tuned joint proposal is more effective. Variations include random or systematic coordinate scans, random-walk Metropolis algorithms, independence proposals, and using Metropolis updates for difficult blocks inside a Gibbs scheme. Both methods generate dependent draws; usable long-run estimates require an appropriate irreducible, aperiodic chain and assessment of mixing, not merely a formal update rule.

For the change-point calculation, put $v={\sigma }^2$ andMultiplication of the Gaussian likelihood function by the specified prior distribution gives the formal posterior distribution kernel, with respect to $d\theta d\varphi dv$ and counting measure on $m$,

There is an empty component under an improper prior at $m=n$. The sum $S_n$ is independent of $\varphi$, whose prior distribution is flat on all of $\mathbb{R}$. For every fixed finite $\theta$ and $v>0$, the kernel is a strictly positive constant in $\varphi$, henceThe configuration has positive prior probability $1/n$. Integrating over, for example, a bounded interval of $\theta$ and $1\le v\le 2$ already makes the joint normalization infinite. Thus the printed prior distribution does not produce a posterior probability distribution. This failure is present for every data set, not just for a special arrangement of observations.

The formal full conditionals expose exactly where the requested sampler breaks. For $m<n$, completing the two mean squares givesThe variance kernel is that of $\mathrm{IG}(n/2,S_m/2)$ when $S_m>0$, and the discrete conditional probabilities are proportional to $e^{-S_m/(2v)}$. At $m=n$ the conditional kernel for $\varphi$ is constant and cannot be normalized; it is not a normal distribution with an infinite variance. Therefore no valid Gibbs sampler, and no value of $P(m=n\mid x)$, exists for the model as printed. A chain that simply substitutes an arbitrary distribution for this missing conditional would be sampling a different model.

Here is a fully specified proper-prior repair retaining the no-change configuration. Choose fixed hyperparametersindependently, with $t_{\theta },t_{\varphi },a_0,b_0>0$. Its Gaussian change-point posterior with proper priors isIt is normalizable even for degenerate data: the likelihood function is at most $(2\pi {)}^{-n/2}{v}^{-n/2}$, and its integral against the variance prior distribution is bounded byThe normal mean priors integrate to one, and the normalizing constant is positive. This proves posterior propriety for the repaired model.

Completing the conditional squares gives the Gaussian change-point Gibbs updates. DefineThe four updates areAt $m=n$, $M_{\varphi }={\mu }_{\varphi },V_{\varphi }=t_{\varphi }$, giving its proper prior distribution unchanged. Starting from any finite means, positive variance and allowed $m$, cycle through these four updates using the latest values. The sum-of-squares terms can be evaluated from cumulative data sums; normalize the discrete log weights after subtracting their maximum to avoid numerical underflow. The proper positive conditionals allow movement among all split values and throughout the continuous parameter support.

Under this specified repair, $P(m=n\mid x)$ is the posterior probability of no change within the observed sequence: all $n$ data points belong to the first mean regime. For $N$ retained draws after initialization has ceased to affect the estimates, its Markov chain Monte Carlo estimator isAlternatively average the displayed conditional probability of $m=n$ at the sampled continuous parameters, giving an estimate by Rao-Blackwellization. Autocorrelation must be accounted for in its Monte Carlo uncertainty. These are probabilities and estimators for the explicitly repaired model; the original unnormalizable kernel supplies neither.

For a regular one-parameter sampling distribution, the Fisher information and Jeffreys prior areUnder the usual differentiation and integrability conditions, $I(\theta )=-{\mathbb{E}}_{\theta }[{\partial }_{\theta }^2\log p_Y(Y\mid \theta )]$. This prior distribution transforms as a density under smooth one-to-one reparameterizations, so the rule is coordinate invariant. Its integral need not be finite; posterior propriety must still be established if it is an improper prior.

Let $\mathcal{D}=(\{{\hat{m}}_s\},\{z_s\})$. Bayes' theorem and the stated prior factorization give the posterior densityThe normalizing constant, when finite and positive, is the integral of this product over its admissible parameter domain. An improper prior requires checking posterior propriety before interpreting this expression as a probability density.

A log-flat improper prior $p(v)\propto 1/v$ is not appropriate when all known $r_s={\sigma }_{m,s}^2$ are positive: the integrated likelihood in (iii) has a positive finite limit as $v\downarrow 0$, so its integral against $dv/v$ diverges. One defensible noninformative choice is the scalar Jeffreys prior for an additive variance component, calculated holding the mean parameters fixed:Indeed the normal variance score has information $I_{vv}=\frac{1}{2}{\sum }_s(v+r_s{)}^{-2}$. This is a scalar conditional-information prior, not a claim that its product with the flat mean prior is the joint Jeffreys prior. It is bounded near zero and is $O(v^{-1})$ at infinity. The integrated likelihood is uniformly bounded near zero and is bounded by a constant times $v^{-(N-1)/2}$ at infinity, independently of the mean-shape parameters, since $e^{-Q/2}\le 1$. Thus $N>1$ and a proper external prior on the physically admissible $(w,{\Omega }_M)$ ensure posterior propriety. Proper weak scale priors are another option. If known errors vanish, the boundary argument and appropriate prior need separate reconsideration.

For an efficient parameterization, use $\theta =(\beta ,u,w,{\Omega }_M)$ with $u=\log v$ and $\beta =M_0+h(H_0)$. A chain on this joint reduced parameter space targetsThe factor $e^u$ is the Jacobian determinant. At each iteration propose ${\theta }^{\prime }=\theta +\epsilon$, with $\epsilon \sim N(0,{\Sigma }_q)$ for a fixed nonsingular proposal covariance, and accept with probability $\min \{1,\tilde{\pi }({\theta }^{\prime })/\tilde{\pi }(\theta )\}$. Proposals outside the physical prior domain have target zero. Tune ${\Sigma }_q$ during warmup and freeze it for the retained random-walk Metropolis algorithm. To obtain samples of the full original parameter vector, independently draw $H_0$ from its retained positive prior and reconstruct $M_0=\beta -h(H_0)$; the resulting vector is $(M_0,v,H_0,w,{\Omega }_M)$. Analytically marginalizing $\beta$ using (iii) would also be valid.

Use several dispersed chains, trace plots, rank-normalized split $\hat{R}$, and the effective sample size of a Markov chain for each parameter and for $w,w^2$. These Markov chain Monte Carlo convergence diagnostics reveal poor mixing and disagreement but do not prove convergence. Estimate the integrated autocorrelation time ${\tau }_{\mathrm{int}}$; with $B$ retained draws, $B_{\mathrm{eff}}\approx B/{\tau }_{\mathrm{int}}$. Default to no thinning of a Markov chain, so the thinning factor is $1$. If storage requires thinning, choose a spacing after inspecting the autocorrelation and verify the retained-chain effective sample size; no finite spacing guarantees independent draws.

Using the retained $w_b$ samples, computeThese are posterior summaries, provided the corresponding moments exist, not uncertainties of the numerical estimates. A proper prior alone does not ensure finite second moments; choose or check the external prior accordingly. The Monte Carlo standard error of the posterior mean is approximately $\widehat{\mathrm{SD}}(w)/\sqrt{B_{\mathrm{eff}}}$.

Write $v={\sigma }_{\mathrm{int}}^2$, $r_s={\sigma }_{O,s}^2$ and $o_s={\hat{O}}_s$. A flat density in $\log \tau$ means $p(\tau )\propto 1/\tau$ with respect to $d\tau$; likewise $p(v)\propto 1/v$. The Jacobian determinant must therefore appear when using the original scale variables. The joint hierarchical Bayesian model kernel, with respect to $d{\mu }_{C}dvd\tau {\prod }_{s}d{o}_{s}d{C}_{s}d{E}_s$, isThis is a kernel, not a normalized joint probability distribution, because the stated priors are improper. Conditioning on $o$ would still require posterior propriety. In fact that check fails here, as shown in part (iii)(c); proper formal full conditionals alone do not repair it. The Gaussian–exponential hierarchical colour model distinguishes the intrinsic normal colour, positive interstellar dust reddening, and measurement noise.

Prior probability describes uncertainty before the current observations are incorporated. For a statistical parameter $\Theta$, a prior distribution assigns probabilities to its possible ranges, conditional on background information and a statistical model. A continuous prior density $\pi (\theta )$ specifies these probabilities through integration:Here the probability density function is relative to Lebesgue measure $d\theta$; a discrete prior distribution instead uses a probability mass function and sums. Density at one point is not the probability of that point. A nonnegative kernel with infinite total mass is an improper prior, whose use requires a separate check of posterior propriety.

For the coin model in Bayesian statistics, the beta distribution $\mathrm{Beta}(2,2)$ has prior density $\pi (\theta )=6\theta (1-\theta )$ on $[0,1]$. Its mean is $1/2$ and its variance is $1/20$. It favors moderate heads probabilities without ruling out values near zero or one. For example,The prior curve in Bayesian posterior makes this preference visible; moment matching for a beta prior explains how to choose a beta distribution from a desired mean and variance.

A prior distribution lets earlier evidence or elicited knowledge contribute to statistical inference when the current sample size is limited. It also exposes assumptions that can be checked by prior elicitation and a prior predictive check. Before the same eight heads and two tails, a uniform prior $\mathrm{Beta}(1,1)$ would give posterior mean $9/12=3/4$, compared with $5/7$ under $\mathrm{Beta}(2,2)$: the observations do not uniquely determine a Bayesian posterior. Even a uniform prior makes a choice of parameter coordinates; its child article explains why changing coordinates can destroy uniformity.

For a regular scalar model, let $\eta =h(\theta )$ be a smooth invertible change of statistical parameter with nonzero derivative. The chain rule makes the score function in the new coordinate equal to the old score function times $d\theta /d\eta$. Therefore the expected Fisher information transforms as $I_{\eta }=I_{\theta }(d\theta /d\eta {)}^2$. Taking its square root gives exactly the absolute change-of-variable factor required for a prior density.

Consequently, computing the Jeffreys prior directly in either coordinate produces the same measure after transformation. Its density is not numerically constant across coordinates, nor must either density be uniform. When the prior distribution is proper, compatible normalization gives identical prior probabilities for corresponding sets; the resulting posterior distributions also transform consistently. For an improper prior the equality concerns measure kernels up to a constant, and posterior propriety still needs checking.