Bayesian statistics

Bayesian statistics expresses uncertainty about an unknown statistical parameter through a probability distribution, then uses Bayes' theorem to update it after observing data. The prior distribution describes uncertainty before the current observations; the likelihood function describes how the statistical model generates those observations at each parameter value; the posterior distribution describes uncertainty conditional on the observations and that model. A physical parameter can have one fixed value while its value remains uncertain to us. Assigning it a probability distribution represents that uncertainty under the chosen model, rather than requiring the physical parameter to change randomly between measurements.

For example, suppose ten Bernoulli trials are independent conditional on an unknown heads probability $\theta$, with a beta distribution prior $\mathrm{Beta}(2,2)$. Eight heads and two tails give the posterior distribution $\mathrm{Beta}(10,4)$ by Beta-binomial conjugacy. Its posterior mean is $10/14=5/7\approx 0.714286$, between the prior mean $1/2$ and observed binomial proportion $8/10$. The Bayesian posterior below computes the whole update, and the credible interval below summarizes its remaining uncertainty. The conjugate update is explained in MIT 18.05, Reading 15: beta and normal conjugate priors.

The useful feature is that one posterior distribution supports several questions: a posterior mean estimates a value, a credible interval reports a range, and a posterior predictive distribution predicts a future observation while averaging over parameter uncertainty. With conditionally independent new data, today's posterior distribution can become tomorrow's prior distribution. These conclusions remain conditional on the statistical model and prior distribution; a precise update does not establish that either assumption fits the experiment.

A historical starting point is Thomas Bayes's essay in the 1763 volume of the Royal Society's Philosophical Transactions, communicated after his death by Richard Price. Its opening problem asks for uncertainty about an unknown event probability given observed successes and failures: the same direction of reasoning illustrated by the coin example. See Bayes and Price, An Essay towards solving a Problem in the Doctrine of Chances, opening letter and problem.

A credible set is a measurable set of parameter values containing specified posterior probability. At credibility level $1-\alpha$, it satisfies ${\int }_C\pi (\theta \mid x)d\theta =1-\alpha$ for a continuous posterior. A credible interval is a credible set that is an interval. Equal-tailed and highest-density choices add selection rules; the posterior-mass requirement alone does not make the set unique. Its probability statement is conditional on the observed data and prior model, whereas a frequentist confidence interval is interpreted through repeated-sample coverage.

If inclusion depends only on fully observed data $d$, then $p(S\mid d,\theta )=p(S\mid d)$ cancels from Bayes' theorem for that individual's parameters. Selection can still matter for unknown population parameters and for missing or unconditioned features. For independent detected observations conditional on their count and population parameters $\Lambda$, each likelihood has normalization $\alpha (\Lambda )=\int S(d)p(d\mid \Lambda )dd$. A model for the detected count adds its own count likelihood. Do not combine a selected-population prior with a truncated-data likelihood in a way that counts the same selection twice.

For data $x$, a positive prior distribution density $\pi$ and likelihood function $\ell$, Bayes' theorem gives posterior density $h(\theta \mid x)=\pi (\theta )\ell (\theta )/m(x)$. Thus $h/\pi$ reconstructs the likelihood up to a constant. Estimate the posterior density from simulation, divide by the known prior density and maximize over the search region. A posterior mode alone instead estimates a maximum a posteriori estimate; it estimates a likelihood maximizer only when the prior is constant there. Prior support must include the likelihood maximizer of interest.

// Target: biology.bigb

BUGS specifies a probabilistic model through stochastic nodes, deterministic nodes and observed data, then uses Markov chain Monte Carlo to simulate its Bayesian posterior. Its normal sampling notation uses a precision parameter rather than a variance, and its gamma notation uses shape and rate.

A zeros trick implements a positive likelihood function $L(\theta )$ by adding an observed zero with Poisson distribution mean $K-\log L(\theta )$. If this mean is nonnegative throughout the parameter support, its likelihood is $e^{-K}L(\theta )$ and yields the intended Bayesian posterior. The constant $K$ must be independent of the parameter; arbitrary clipping changes the target.

Variational inference approximates a posterior distribution by minimizing a Kullback-Leibler divergence over a chosen family of probability distributions. For an unnormalized posterior density $h$ and a trial probability density function $q$, minimizing ${\mathbb{E}}_q[\log q-\log h]$ is equivalent to minimizing $\mathrm{KL}(q\Vert h/\int h)$.

Under fixed-support regularity permitting differentiation under the integral sign, apply the product rule to $\int s_{\lambda }{W}_{\lambda }$. The first derivative term is $\int W_{\lambda }\nabla s_{\lambda }=\mathbb{E}[W_{\lambda }\nabla \log s_{\lambda }]$ and the second is $\mathbb{E}[\nabla W_{\lambda }]$. This proves the displayed identity. For an evidence lower bound, take $W_{\lambda }=\log p(t_{\lambda }(\epsilon ),x)-\log q_{\lambda }(t_{\lambda }(\epsilon ))$, holding noise $\epsilon$ fixed in the inner derivatives. Parameter-independent continuous noise contributes a reparameterization gradient; untransformed discrete choices contribute a score-function gradient estimator. A mixed base law cannot be called parameter independent if its discrete factors retain variational parameters.^[ref]

A guide program is a probabilistic program whose generated probability distribution approximates a model's posterior distribution. Its adjustable parameters are fitted by variational inference. The guide must assign probability only where the target allows it when minimizing $D_{\mathrm{KL}}(q\Vert p)$, but a tractable guide can still miss important target regions.

A reparameterization gradient differentiates an expected value by representing the simulated random variable as a differentiable transformation of parameter-independent noise. For $X_{\eta }=g_{\eta }(\epsilon )$, the identity ${\nabla }_{\eta }\mathbb{E}[F_{\eta }(X_{\eta })]=\mathbb{E}[{\nabla }_{\eta }{F}_{\eta }(g_{\eta }(\epsilon ))]$ requires justified differentiation under the integral sign. For a normal distribution, use $X=m+e^{\rho }\epsilon$ with $\epsilon$ drawn from the standard normal distribution.

Markov chain variational inference optimizes an initial parametric probability distribution ${\nu }_{\eta }$ by minimizing $\mathrm{KL}(K^t{\nu }_{\eta }\Vert \pi )$ after a fixed number of Markov kernel updates. The best initial probability distribution can depend on $t$; Kullback-Leibler divergence contraction does not imply that optimization commutes with applying the Markov kernel.

Mean-field variational inference restricts the trial probability density function to $q(x)={\prod }_i{q}_i(x_i)$. With the other factors fixed, $q_j^{\ast }(x_j)\propto \exp ({\mathbb{E}}_{q_{-j}}\log \pi (x_j,X_{-j}))$, provided this expression has a finite positive normalization constant. Subtracting the objective at $q_j^{\ast }$ leaves $\mathrm{KL}(q_j\Vert q_j^{\ast })\ge 0$, proving the update is optimal when the terms are well defined.

Data augmentation enlarges a posterior distribution with latent variables whose removal by marginalization recovers the original posterior distribution. Convenient full conditional distributions then permit a Gibbs sampler on the enlarged space.

A hierarchical Bayesian model assigns distributions to local latent variables conditional on shared hyperparameters, then distributions to observations conditional on those latent variables. Its probabilistic graphical model records this conditional factorization, often using a plate for repeated observations. Integrating out latent variables gives the marginal likelihood of the hyperparameters. Priors described as flat on logarithms require a Jacobian determinant when densities are written in the original positive variables.

Unknown hospital event probabilities can be modelled as exchangeable random variables conditional on shared hyperparameters, for example $P_j\mid m,\kappa \sim \mathrm{Beta}(m\kappa ,(1-m)\kappa )$ independently. Different sample sizes affect the binomial observations but do not by themselves invalidate exchangeability of the risks. Known systematic covariate differences, such as case mix or procedure type, should instead enter the risk model; only residual effects may then be exchangeable. Exchangeability permits heterogeneous realized risks and is not an assertion that all risks are equal.

// Target: biology.bigb

A normal local effect ${\alpha }_i\mid \delta ,\tau \sim N(\delta ,{\tau }^2)$ can be represented by a standardized latent variable $z_i\sim N(0,1)$ independent of the hyperparameters, with ${\alpha }_i=\delta +\tau z_i$. This preserves the hierarchical model while changing the coordinates explored by Markov chain Monte Carlo. It often improves mixing when individual effects are weakly informed, reducing strong scale-effect dependence. Centered coordinates can be better with very informative individual data. The stochastic prior on $z_i$ implements the transformation correctly; re-expressing an existing density directly instead requires the appropriate Jacobian determinant.

Partial pooling estimates related local parameters jointly through shared hyperparameters. Each local Bayesian posterior balances its own likelihood function against a group prior distribution, with stronger shrinkage when its data are weak or the group variation is small. It lies between forcing all local parameters to be equal and estimating them independently. Random-effects meta-analysis is an important application.

A Gamma–Poisson hierarchical model uses $Y_i\mid {\theta }_i\sim \mathrm{Poisson}(t_i{\theta }_i)$, ${\theta }_i\mid b\sim \mathrm{Gamma}(a,b)$, and $b\sim \mathrm{Gamma}(c,r)$, with gamma distributions in shape-rate convention. Conditional independence gives full conditional distributions ${\theta }_i\mid b,y\sim \mathrm{Gamma}(a+y_i,b+t_i)$ and $b\mid \theta ,y\sim \mathrm{Gamma}(c+na,r+{\sum }_i{\theta }_i)$.

For $Y_i\mid {\theta }_i\sim \mathrm{Poisson}({\theta }_i)$ and ${\theta }_i\mid \psi \sim \mathrm{Exp}(\psi )$, a scale-invariant prior $d\psi /\psi$ induces the Haldane prior on $\varphi =(1+\psi {)}^{-1}$. Integrating out each local mean gives $p(y_i\mid \varphi )=(1-\varphi ){\varphi }^{y_i}$. Thus $\varphi \mid y\sim \mathrm{Beta}(s,n)$ for $s={\sum }_i{y}_i>0$. The gamma full conditional distribution has mean $(y_i+1)\varphi$, so the tower property gives $(y_i+1)s/(s+n)$, equal to the displayed partial pooling formula. For $s=0$, the posterior is improper and this expectation is undefined.

For independent Poisson distributions with means $u{e}^{{\eta }_k}$ and a shape–rate Gamma distribution prior on $u$, integrating $u$ adds $a$ to the total-count exponent and $b$ to the sum of relative rates. This distinguishes gamma mixing from conditioning on the total count, which gives a multinomial distribution.

An observed astronomical colour is the sum of a normal intrinsic colour, a nonnegative dust contribution with an exponential distribution, and normal measurement noise. With latent $C_s\sim N(\mu ,v)$, $E_s\sim \mathrm{Exp}(\text{mean }\tau )$ and $O_s\mid C_s,E_s\sim N(C_s+E_s,r_s)$, the Gibbs sampler conditionals for $C_s$ are normal and those for $E_s$ are truncated normal distributions. A flat prior on $\mu$ and inverse-gamma priors on $v,\tau$ give inverse-gamma conditional updates for the two scales. Log-flat priors on both scales instead make the joint posterior improper when all $r_s>0$, despite formally proper full conditionals at generic latent states: the marginal observed likelihood has a positive limit at either zero-scale boundary. Proper positive-scale priors restore a genuine joint posterior.

Maximum a posteriori estimation selects a parameter value maximizing its posterior density relative to a specified reference measure. Its output is a maximum a posteriori estimate. The maximizing value need not be invariant under a nonlinear reparameterization, because a density transforms with a Jacobian.

A maximum a posteriori estimate maximizes the posterior density of a parameter or latent configuration. Equivalently, it maximizes the sum of the log likelihood and log prior, relative to the chosen reference measure.

A credible interval is an interval $C(d)$ containing a prescribed amount $1-\alpha$ of posterior probability for an unknown scalar statistical parameter $\Theta$. For a continuous posterior distribution with posterior density $p(\theta \mid d)$, its endpoints satisfyThe conditioning includes the chosen statistical model and prior distribution. For a discrete posterior distribution, exact equality may be impossible because of its probability masses; an interval can instead be required to contain at least the requested posterior probability. An interval compresses the full posterior distribution into a range whose uncertainty can be communicated in the parameter's own units. See MIT 18.05, Class 16: probability intervals.

For the eight-head, two-tail example in Bayesian statistics, the posterior density is $2860{\theta }^9(1-\theta {)}^3$ on $[0,1]$. Its cumulative distribution function, obtained by integration of this polynomial, isSolving $F(a)=0.025$ and $F(b)=0.975$ by the bisection method givesThese are numerical approximations to the exact quantiles defined by those equations; the endpoints were computed from the cumulative distribution function, rather than read off a plot. The resulting equal-tailed credible interval leaves 2.5% posterior probability on each side. Its shaded area is 95% of the total area under the posterior density.

Credible intervals are not unique. The one-sided credible interval $[0,F^{-1}(0.95)]\approx [0,0.887334219]$ also has 95% posterior probability, but leaves all 5% outside it on the right. A highest density region instead selects points above a density threshold; it need not be an equal-tailed credible interval, and when the posterior density has separated peaks it need not be one interval at all. Thus a reported credible interval should specify its construction.

A 95% credible interval makes a statement about the statistical parameter conditional on these observed data and the Bayesian model. A 95% confidence interval instead refers to the coverage probability of a procedure under repeated sampling at a fixed parameter value. These are different conditioning statements: 95% posterior probability does not by itself imply 95% coverage probability for every fixed parameter. The confidence interval article develops that sampling interpretation.

A highest posterior density interval is a credible interval whose included points have at least as high posterior density as excluded points, apart from boundary and null-set choices. It is a highest density region of prescribed posterior probability that happens to be an interval. For a continuous unimodal posterior with two interior endpoints, the endpoint densities agree. A monotone posterior on a half-line instead starts at a support boundary, as in a Pareto highest density interval. For a multimodal posterior distribution, the highest-density region can be disconnected and cannot always be represented by a single interval. Density-level regions use a specified parameter coordinate and reference measure; they are not generally preserved by nonlinear reparametrization.

For a beta distribution with shape parameters $A,B>1$, the probability density function vanishes at zero and one and is strictly increasing up to $m=(A-1)/(A+B-2)$ and strictly decreasing afterwards. This follows because the derivative of its logarithm is $(A-1)/t-(B-1)/(1-t)$, whose sign changes at $m$.

For each $0<\alpha <1$, the highest posterior density interval when this beta distribution is a posterior distribution has unique endpoints $0<l<m<u<1$ determined byThe second equation gives equal endpoint posterior densities. For every density threshold strictly between zero and the modal height there is precisely one crossing on each side. The enclosed probability is continuous and strictly decreasing from one to zero as the threshold increases, proving existence, uniqueness and the highest posterior density interval property. The restrictions $A,B>1$ are essential to this two-interior-endpoint characterization; monotone cases use a support boundary instead.

An equal-tailed credible interval uses quantiles of a continuous posterior distribution to leave posterior probability $\alpha /2$ in each tail. It is also called a central credible interval; central refers to equal tail probabilities, not equal distances from the posterior mean. For the beta distribution $\mathrm{Beta}(10,4)$ in credible interval, the 95% endpoints are approximately $0.461868461$ and $0.909079605$, whose midpoint $0.685474033$ differs from the posterior mean $5/7$.

A one-sided credible interval puts all excluded posterior probability in one tail. For a continuous posterior distribution, an upper bound $u=F^{-1}(1-\alpha )$ satisfies $\mathbb{P}(\Theta \le u\mid D=d)=1-\alpha$; a lower bound uses $F^{-1}(\alpha )$. Intersecting with the parameter's allowed range removes points of zero posterior probability. For the beta distribution $\mathrm{Beta}(10,4)$, the upper 95% bound is $u\approx 0.887334219$, so $[0,u]$ is the alternative credible interval shown above.

An empirical Bayes method estimates prior hyperparameters from the marginal distribution of the observed data and then uses the resulting fitted prior in Bayesian inference.

A maximum marginal likelihood estimator chooses a hyperparameter by maximizing the data density obtained after integrating the model parameter against its prior distribution.

A hyperparameter controls a family of prior distributions or statistical models and is fixed, estimated, or assigned a further prior at a higher level of a hierarchical model.

A Dirichlet process is a distribution over probability measures such that the masses assigned to every finite measurable partition have a Dirichlet distribution. Its draws are almost surely discrete.

A Dirichlet prior with parameters $({\alpha }_1,\dots ,{\alpha }_K)$ combined with categorical labels having counts $(N_1,\dots ,N_K)$ gives a Dirichlet posterior with parameters $({\alpha }_1+N_1,\dots ,{\alpha }_K+N_K)$.

Prior probability describes uncertainty before the current observations are incorporated. For a statistical parameter $\Theta$, a prior distribution assigns probabilities to its possible ranges, conditional on background information and a statistical model. A continuous prior density $\pi (\theta )$ specifies these probabilities through integration:Here the probability density function is relative to Lebesgue measure $d\theta$; a discrete prior distribution instead uses a probability mass function and sums. Density at one point is not the probability of that point. A nonnegative kernel with infinite total mass is an improper prior, whose use requires a separate check of posterior propriety.

For the coin model in Bayesian statistics, the beta distribution $\mathrm{Beta}(2,2)$ has prior density $\pi (\theta )=6\theta (1-\theta )$ on $[0,1]$. Its mean is $1/2$ and its variance is $1/20$. It favors moderate heads probabilities without ruling out values near zero or one. For example,The prior curve in Bayesian posterior makes this preference visible; moment matching for a beta prior explains how to choose a beta distribution from a desired mean and variance.

A prior distribution lets earlier evidence or elicited knowledge contribute to statistical inference when the current sample size is limited. It also exposes assumptions that can be checked by prior elicitation and a prior predictive check. Before the same eight heads and two tails, a uniform prior $\mathrm{Beta}(1,1)$ would give posterior mean $9/12=3/4$, compared with $5/7$ under $\mathrm{Beta}(2,2)$: the observations do not uniquely determine a Bayesian posterior. Even a uniform prior makes a choice of parameter coordinates; its child article explains why changing coordinates can destroy uniformity.

A class prevalence applicable before considering the current case-specific evidence. Under random sampling from the relevant reference population, it sets the prior class probability used in Bayes' theorem.

The prior mean is the expected value of a statistical parameter under its prior distribution, when this expected value exists. The posterior mean instead uses the posterior distribution after conditioning on observations.

Translate external information into a prior distribution by asking for representative values, uncertainty intervals and tail probabilities. Fit parameters of a suitable distribution to those judgments and inspect the resulting prior predictive distribution. For a probability believed near $0.1$, a beta distribution with parameters $0.1\kappa ,0.9\kappa$ has the desired mean; its concentration $\kappa$ can be chosen to match a stated probability of exceeding $0.15$. The words “near” and “unlikely” alone do not specify one unique prior.

For two exhaustive mutually exclusive models of positive prior probability, prior odds compare their probabilities before observing the data. They must be distinguished from the parameter prior distributions within those models. A Bayes factor multiplies these prior odds to produce posterior odds.

A prior density is the probability density function of a continuous prior distribution with respect to a specified reference measure. A proper prior density is nonnegative and integrates to one. Prior-density factors enter Bayesian posterior and RJ-MCMC acceptance formulas along with the likelihood and model prior probabilities.

A continuous spike-and-slab prior mixes a narrow continuous probability distribution around zero with a much wider component. It permits practical effect selection while retaining a continuous posterior density. Its narrow component is not the exact-zero mass of a point-null mixture prior. A latent component indicator aids computation; a wide-component draw can still have small magnitude, so posterior effect thresholds and component membership answer different questions.

A uniform prior on a finite-volume parameter region assigns constant probability density there and zero outside. On $[0,1]$ it is $\mathrm{Beta}(1,1)$, giving a beta distribution posterior after binomial distribution data. Uniformity depends on the coordinate: a nonlinear reparametrization introduces a Jacobian and generally does not preserve a uniform density.

The Bayesian posterior is the conditional probability distribution of an unknown statistical parameter after observing data. Suppose a continuous prior distribution has prior density $\pi (\theta )$ relative to Lebesgue measure $d\theta$, and $p(d\mid \theta )$ is the sampling probability mass function or probability density function of the observed data $d$ in the statistical model. Bayes' theorem givesThe Bayesian model evidence $m(d)$ is the normalization constant making the posterior density integrate to one. The strict positive, finite condition is essential: a zero or infinite normalizer does not define this posterior distribution, even if the unnormalized expression can be written down. Other reference measures give the corresponding density formula; discrete parameter values replace the integral by a sum. The posterior propriety article gives examples where normalization fails.

With $d$ fixed, $L(\theta ;d)=p(d\mid \theta )$ is a likelihood function of $\theta$. Its normalization as a sampling distribution is over possible data, not over parameter values. In particular, a likelihood function is not a probability density function for $\Theta$ and need not integrate to one over $\theta$. The prior density supplies the probability weighting, and the Bayesian model evidence normalizes their product. This distinction prevents a tempting mistake: reversing the conditioning merely by exchanging the words “data” and “parameter”.

For eight heads among ten conditionally independent Bernoulli trials, take the observed data to be the heads count $D=8$. Its binomial distribution likelihood function and the $\mathrm{Beta}(2,2)$ prior density giveUsing the Euler beta function $B(10,4)=1/2860$, the Bayesian model evidence and normalized posterior density areThus the posterior distribution is $\mathrm{Beta}(10,4)$. If the data are instead one particular ordered sequence with eight heads and two tails, its likelihood function omits the factor 45 and its Bayesian model evidence becomes $3/1430$. The factor cancels in Bayes' theorem, leaving the same posterior distribution; these two data descriptions have different sampling probabilities but identical information about $\theta$ here.

The prior density and posterior density use the left axis and each integrate to one. The dashed likelihood function uses the right axis and is divided by its maximum solely to compare its shape; it is not a parameter probability density function. The likelihood function peaks at $8/10$, while the posterior mode is $9/12=3/4$ and the posterior mean is $5/7$. Multiplying by the prior density shifts the preferred values toward one half while preserving a range of possibilities.

The resulting posterior variance is $10\cdot 4/(14^2\cdot 15)=2/147\approx 0.0136054$, compared with prior variance $1/20$. In this example the update reduces uncertainty. The posterior predictive probability of heads on the next conditionally independent toss is $\mathbb{E}[\Theta \mid D=8]=5/7$, and the credible interval above reports a 95% range. These are summaries of the same posterior distribution, rather than competing estimates of one density peak.

For independent and identically distributed observations from a normal distribution with known variance ${\sigma }^2$, an exponential distribution prior of rate $\lambda$ on their nonnegative mean gives this truncated normal distribution posterior. Complete the square in $-n(\mu -\bar{x}{)}^2/(2{\sigma }^2)-\lambda \mu$; the truncation remains essential even when the untruncated normal mean is negative.

A posterior moment is a moment computed under a posterior distribution. Its first raw moment is the posterior mean; its second central moment is the posterior variance.

For one observation from a continuous uniform distribution on $[0,\theta ]$ and prior density $\theta e^{-\theta }$, the likelihood cancels the factor $\theta$ and leaves $e^{-\theta }{1}_{\{\theta \ge x\}}$. Normalization gives posterior density $e^{-(\theta -x)}{1}_{\{\theta \ge x\}}$, for $x\ge 0$. Its mean is $x+1$ and variance one. The Bayes estimator under squared error loss is therefore $x+1$, unchanged by multiplying the loss by a positive constant.

For independent observations uniform on $(\theta -d,\theta +d)$ and a uniform prior on $(A,B)$, the likelihood is constant wherever all data-compatible intervals overlap. The posterior is therefore uniform on the displayed intersection, provided it has positive length. Both squared-error and absolute-error Bayes estimates are its midpoint. An empty or zero-length interval has zero marginal likelihood and cannot be normalized into this ordinary posterior density.

For one uniformly sampled integer $Y\in \{1,\dots ,N\}$, the likelihood function for the unknown endpoint is $N^{-1}{1}_{N\ge y}$. A flat improper prior on positive integers gives an improper posterior distribution, since the harmonic series diverges. A reciprocal prior instead gives a proper posterior proportional to $N^{-2}$ for $N\ge y$, with approximate survival probability $y/n$ and finite quantiles, but infinite posterior mean. A uniform prior cut off at $M$ gives posterior mean $(M-y+1)/(H_M-H_{y-1})\sim M/\log M$, demonstrating sensitivity to that arbitrary cutoff.

For continuously distributed uncertain quantities, define rank one to be the largest. A joint posterior distribution induces a discrete distribution on each quantity's rank. Estimate it by ranking the whole vector in each joint draw and averaging rank indicators; the probability of rank one is the probability of being largest. Ranking posterior means does not recover this uncertainty. The joint draws must preserve any posterior dependence, and tie conventions must be specified for models with atoms.

A posterior variance is the variance of a parameter under its Bayesian posterior, namely $\mathbb{E}[(\theta -\mathbb{E}[\theta \mid y]{)}^2\mid y]$ when finite. It describes remaining uncertainty after observing data and is distinct from the repeated-sampling variance of a fitted estimator.

For a vector parameter with finite second moments and posterior mean $m_y$, the posterior covariance matrix records its uncertainty and linear dependence after conditioning on the observations. Its diagonal entries are posterior variances. In a regular locally flat-prior normal distribution approximation it is the inverse observed information. A quadratic form has posterior expectation equal to the form at the posterior mean plus the trace of its matrix times this covariance.

The distribution of a new or missing quantity integrates its model over the Bayesian posterior: $p(y^{\ast }\mid y)=\int p(y^{\ast }\mid \varphi ,y)p(\varphi \mid y)d\varphi$. Both parameter and residual uncertainty should enter predictive draws used for multiple imputation.

If future Bernoulli random variables are conditionally independent given a shared parameter $\Theta$, with success probability $\pi (\Theta )$, their all-success conditional probability is $\pi (\Theta {)}^k$. Taking a posterior expectation proves the identity. For $k\ge 1$, Jensen's inequality and $0\le \pi \le 1$ give $(\mathbb{E}[\pi \mid D]{)}^k\le \mathbb{E}[{\pi }^k\mid D]\le \mathbb{E}[\pi \mid D]$. For two distinct future trials the posterior covariance is $\mathbb{E}[{\pi }^2\mid D]-(\mathbb{E}[\pi \mid D]{)}^2=\mathrm{Var}(\pi \mid D)$. Thus shared parameter uncertainty can induce positive dependence even when trials are conditionally independent.

A posterior predictive check compares a summary of the observed data with the same summary in replicated data drawn from a model's posterior predictive distribution. The replication must match the target: replicating an existing group's observations conditional on its effect differs from predicting a new group with a newly drawn effect. Such checks expose mismatches in dispersion, tails or other features; using the observations to fit and check the model means they are not automatically calibrated frequentist tests.

In a hierarchical Bayesian model, replication conditional on fitted observation-level random effects checks the sampling layer. Replication with fresh random effects drawn from their population law also checks the population mean and variation. To assess a dose-response curve, sample the global parameters from their posterior distribution, draw fresh effects for new plates at each existing dose, and compare replicated means, spreads and trend discrepancies with the observed values. Reusing the fitted plate effects can conceal a misspecified mean curve by reproducing deviations those effects already absorbed.

For independent observations $y_s\sim N(\beta +d_s,V_s)$, put $a_s=V_s^{-1}$, $S={\sum }_s{a}_s$, $r_s=y_s-d_s$, $\bar{r}=S^{-1}{\sum }_s{a}_s{r}_s$ and $Q={\sum }_s{a}_s(r_s-\bar{r}{)}^2$. Integrating the likelihood function against a flat improper prior on $\beta$ gives, up to the arbitrary prior constant,Indeed ${\sum }_s{a}_s(r_s-\beta {)}^2=Q+S(\beta -\bar{r}{)}^2$, and the remaining one-dimensional Gaussian integral is $\sqrt{2\pi /S}$. The conditional Bayesian posterior of $\beta$ is $N(\bar{r},S^{-1})$.

A posterior predictive probability averages a future-event probability over the Bayesian posterior. For a Bernoulli distribution with unknown parameter $p$, it is $\mathbb{E}[p\mid \mathcal{D}]$. Under Beta-binomial conjugacy with posterior $\mathrm{Beta}(a,b)$, the predictive success probability is $a/(a+b)$.

A posterior density is the density of a posterior distribution with respect to a chosen reference measure.

The log-posterior is the logarithm of a positive posterior density. Its gradient is the posterior score control variate when boundary terms vanish. An additive constant independent of the parameter does not affect optimization or differentiation.

The posterior mean is the expected value of a parameter under its posterior distribution; under squared-error loss it is the Bayes estimator.

Write the log posterior kernel as $H_n(\theta )=\log p(\theta )+{\sum }_i\log f(Y_i\mid \theta )$, with a unique concentrating interior mode $\tilde{\theta }$ and $J_n=-H_n^{\prime \prime }(\tilde{\theta })>0$. Apply Laplace's method to the numerator and denominator of the expected value under the posterior distribution. Their common leading Gaussian factor cancels. Under smoothness and localization, the first correction is $g^{\prime \prime }(\tilde{\theta })/(2J_n)+g^{\prime }(\tilde{\theta })H_n^{\prime \prime \prime }(\tilde{\theta })/(2J_n^2)$, of order $n^{-1}$ when the posterior curvature and log-kernel derivatives have their usual order $n$.

A posterior probability is the probability assigned to an event by the posterior distribution after conditioning on the observed data.

For two exhaustive mutually exclusive models, posterior odds are the ratio of their posterior probabilities. Bayes' theorem givesThe second factor is the Bayes factor, formed from model prior predictive distributions. Evidence favouring one model need not make that model's posterior probability exceed one half if its prior probability was sufficiently small.

When the prior odds of a real effect are $R$, a study's Type I error is $\alpha$, and its statistical power is $1-\beta$, a positive report updates those odds to $R(1-\beta )/\alpha$. This follows directly from Bayes' theorem. A small false-positive rate does not itself make the effect's posterior probability high: the prior odds and power also matter. The posterior exceeds one half exactly when $R>\alpha /(1-\beta )$.

A point-null mixture prior assigns positive mass to one parameter value and distributes the remaining mass continuously over alternatives. Posterior point mass is obtained by dividing the null component of the marginal density by the full marginal density.

For $\overline{X}\mid \mu \sim N(\mu ,1/n)$ and an equal mixture of $\mu =0$ and $\mu \sim N(0,{\tau }^2)$,

For the equal point-null mixture,

With a fixed diffuse alternative prior, a frequentist p-value and the posterior probability of a point null can order evidence differently. For $n=100$ and $\tau =1$, the p-value exceeds the null posterior near zero, while sufficiently far in the tails the null posterior exceeds the p-value because it has a slightly slower Gaussian decay.

An improper prior is a nonnegative prior kernel with infinite total mass. It can still produce a proper posterior after multiplication by the likelihood and normalization.

The Haldane prior has formal beta distribution parameters $(0,0)$. It is an improper prior, with infinite mass at both endpoints of $(0,1)$. For a binomial distribution likelihood with $s$ successes and $f$ failures, its posterior is $\mathrm{Beta}(s,f)$ only when $s,f>0$. A density flat in the log odds induces this kernel by the Jacobian determinant $d\log [p/(1-p)]/dp=1/[p(1-p)]$.

The measure $d\sigma /\sigma$ is invariant under multiplication by every positive constant. In the logarithmic coordinate it is Lebesgue measure. It cannot be normalized over $(0,\infty )$, although a bounded truncation is a log-uniform distribution.

A random-effect model with positive marginal likelihood at scale zero cannot yield a proper posterior distribution under a prior proportional to the reciprocal scale near zero. On a compact set of the other parameters, continuity and positivity give a positive likelihood lower bound, while ${\int }_0^{\epsilon }d\lambda /\lambda =\infty$. Individual full conditional distributions can still be proper, so their existence alone does not justify a Gibbs sampler with a probability distribution as its joint target.

A posterior kernel $L(\theta )\pi (\theta )$ defines a posterior distribution exactly when its integral is finite and nonzero. Proper full conditional densities do not suffice. For example, if a known positive measurement variance makes a marginal normal likelihood approach a strictly positive limit as a latent variance $v$ tends to zero, the log-flat prior $dv/v$ gives infinite posterior mass at that boundary. The same problem occurs for a latent exponential scale $\tau$ with $d\tau /\tau$ when the observed likelihood has a positive limit as the exponential contribution vanishes.

If a model configuration leaves a parameter absent from its likelihood function, giving that parameter an independent flat improper prior makes the posterior distribution normalization infinite whenever the configuration has positive prior distribution mass. Integrate first over that parameter: a positive constant likelihood function factor is repeated over all of $\mathbb{R}$. This occurs for the second segment mean of a Gaussian change-point detection model when the proposed split is after the final observation. There is no posterior distribution or valid Gibbs sampler for that joint kernel. A proper prior distribution on the unused parameter is necessary if the no-change configuration is retained.

For conditionally independent Poisson observations of mean $\theta$ and a gamma distribution prior of shape $r>0$ and rate $\lambda >0$, the likelihood function contributes ${\theta }^{{\sum }_i{x}_i}{e}^{-n\theta }$. Multiplying by the prior density gives a posterior distribution with shape $r+{\sum }_i{x}_i$ and rate $\lambda +n$. The single-observation update is shape $r+X$ and rate $\lambda +1$.

The posterior mean is the convex combinationThus the prior mean $r/\lambda$ and the sample maximum-likelihood estimate $\bar{x}$ bracket it. Integrating the next conditional Poisson distribution against this posterior distribution gives the Poisson-gamma mixture, a negative binomial distribution; its spread includes uncertainty in $\theta$ as well as the conditional count variation.

If the positive baseline rate has a gamma distribution of shape $a$ and rate $b$, its logarithm has this probability density function with respect to $d\mu$. The extra factor $e^{\mu }$ is the Jacobian determinant of the transformation. After a Gibbs sampler draws the conditional baseline rate, taking its logarithm is an exact update of the intercept. Omitting the Jacobian changes the prior.

Given $\Theta$, independent observations $Y_j$ have Poisson distribution with means $m_j\Theta$ for known exposures. A gamma distribution prior of shape $\alpha$ and rate $\beta$ yields a gamma posterior with shape $\alpha +{\sum }_j{Y}_j$ and rate $\beta +{\sum }_j{m}_j$. The likelihood kernel is ${\theta }^{{\sum }_j{Y}_j}{e}^{-\theta {\sum }_j{m}_j}$. Thus the posterior mean depends on total claims and total exposure, not the number of periods alone.

A gamma prior with shape $\alpha$ and rate $\beta$ combined with $n$ independent observations from an exponential distribution of rate $\theta$ gives the posteriorUnder squared-error loss, the Bayes estimator is its posterior mean $(\alpha +n)/(\beta +{\sum }_i{X}_i)$.