Bayesian model evidence ($\mathcal{Z}$)

The Bayesian model evidence is the prior predictive distribution evaluated at the data $D$. For a statistical model $H_j$ with a normalized prior density ${\pi }_j$, its marginal likelihood isThis is an expected value of the likelihood function under the prior distribution; for discrete statistical parameters, replace the integral by a sum. For continuous observations, $m_j(D)$ is a probability density function value with respect to a specified data reference measure, rather than a probability of one exact observation. Comparisons must use the same data and reference measure. The same $m_j(D)$ normalizes the within-model Bayesian posterior and supplies the between-model comparison in a Bayes factor.^[ref]

A normalized prior distribution makes the Bayesian model evidence an average, so it cannot exceed the largest likelihood function value on its support. A broad prior distribution can assign little mass to values that predict the data well, even when one value gives an excellent fit. The worked uniform-alternative binomial Bayes factor below makes the difference between averaging and maximum likelihood estimation explicit.

For a normalized uniform prior on $[a,b]$, Bayesian model evidence is $Z=(b-a{)}^{-1}{\int }_a^b\mathcal{L}(\theta )d\theta$. It averages the likelihood over the prior range. Extending a well-placed interval into low-likelihood regions reduces that average, but moving or widening a poorly placed interval can improve it. A model prior is separate: posterior model odds equal the evidence ratio multiplied by prior model odds.

If $\mathcal{L}(\theta )=C{e}^{-|\theta -\mu |/s}$, integrate the two exponential branches at $\mu$. An interval of half-width $w$ centred at $\mu$ has $Z/C=(s/w)(1-e^{-w/s})$. An interval entirely below the peak has $Z/C=s[e^{(b-\mu )/s}-e^{(a-\mu )/s}]/(b-a)$. These expressions show quantitatively how fit location and prior width combine.

A prior predictive check simulates parameters from the prior distribution and observations from their sampling model, then examines whether those observations are plausible before conditioning on data. This reveals unintended information from apparently broad priors, especially after nonlinear transformations or when scales and correlations are coupled. It differs from a posterior predictive check, which uses the posterior distribution.

Let $\pi$ be a proper prior density, let $L>0$ for prior-almost every parameter, and let $Z=\int L\pi \in (0,\infty )$. For posterior draws ${\theta }_i$, the harmonic mean estimator isThe identity ${\mathbb{E}}_{\pi (\cdot \mid y)}[1/L]=1/Z$ follows by cancelling $L$ in the posterior density. The variance of $1/L$ is often infinite, making the estimate unstable even when an ergodic average is consistent.^[ref]

The support qualification matters. If $a={\int }_{\{L>0\}}\pi <1$, cancellation instead gives $a/Z$. For a uniform prior on $[0,1]$ and $L(\theta )=1_{[0,1/2]}(\theta )$, the evidence is $1/2$, but every posterior likelihood value is one and the displayed estimator is always one. An improper prior also leaves the absolute evidence normalization undefined.

A Bayes factor measures how data change the relative support for two specified statistical models. Write $m_j(D)=p(D\mid H_j)$ for their Bayesian model evidence and define the direction explicitly:For positive finite $m_0(D),m_1(D)$, a value above one favors $H_1$ over $H_0$; a value below one favors $H_0$ over $H_1$. Reversing the comparison gives $B_{01}={\mathrm{BF}}_{01}=1/{\mathrm{BF}}_{10}$, the opposite direction used in some formulas below.

For mutually exclusive, exhaustive $H_0,H_1$ with positive prior probabilities, divide their two Bayes' theorem formulas. Their common data normalizer cancels, leavingThe Bayes factor therefore multiplies prior odds into posterior odds. It is evidence conditional on the specified statistical models and their within-model prior distributions; it is not $\mathbb{P}(H_1\mid D)$. Model prior probabilities supply a separate input.^[ref]

For an original stipulated example, consider ten independent and identically distributed Bernoulli trials, with heads as success. Compare $H_0:p=0.4$ with $H_1:p=0.7$, both fixed before seeing the data. Take $D$ to be one particular ordered sequence containing eight heads and two tails. The likelihood function under either point model is $p^8(1-p{)}^2$, soIf $\mathbb{P}(H_1)=0.1$, the prior odds for $H_1$ against $H_0$ are $1/9$. Hence the posterior odds are $21.9909706/9\approx 2.4434412$, andEqual model prior probabilities would instead give ${\mathrm{BF}}_{10}/(1+{\mathrm{BF}}_{10})\approx 0.956505$. The same Bayes factor can thus accompany different posterior probabilities. These statements condition on the two-model comparison; they do not establish that either statistical model describes a real coin.

Using the heads count $K=8$ instead of the ordered sequence changes the sample space. The binomial distribution assigns it mass $\left(\genfrac{}{}{0ex}{}{10}{8}\right)p^8(1-p{)}^2$. The same binomial coefficient multiplies both models' Bayesian model evidence and cancels in their Bayes factor. Mixing an ordered-sequence numerator with a count denominator would introduce an erroneous factor of $45$.

Fix the candidate statistical models, their parameter prior distributions, and model prior probabilities before using the current data for this update. The Bayes factor is useful because it separates evidence from model prior odds, but it still depends on the parameter prior distributions. An improper prior has an arbitrary multiplicative normalization: multiplying its kernel by $c$ multiplies its formal marginal likelihood by $c$, even if the within-model posterior distribution is normalizable. Without a separately justified convention, the ordinary Bayes factor is then undefined up to arbitrary constants.^[ref]

For a random sample $X_i\mid \theta \sim N(\theta ,{\sigma }^2)$ with known $\sigma >0$ and prior distribution $\theta \sim N(0,{\tau }^2)$ with $\tau >0$, compare $H_0:\theta <0$ to $H_1:\theta \ge 0$. The prior odds are one by symmetry. Normal-normal conjugacy gives a normal posterior distribution withIts negative and nonnegative masses are $\Phi (-z)$ and $\Phi (z)$, whereSince the Bayes factor is the posterior odds divided by the prior odds,Here $\Phi$ is the cumulative distribution function of the standard normal distribution. The ratio decreases with the sample mean. For any threshold $b>0$, the inequality $B_{01}<b$ is equivalent to $\Phi (z)>1/(1+b)$ and hence toFor $b=1/10$, the quantile is ${\Phi }^{-1}(10/11)\approx 1.33518$, and the equivalent posterior probability threshold for $H_0$ is $1/11$. The symmetry and proper normalization of the two hypothesis-conditional priors are essential to replacing the Bayes factor by the posterior odds directly.

For two models with finite positive prior odds, a sequence of data with identical conditional likelihoods under both models has Bayes factor one. The posterior odds therefore equal the prior odds. This requires equality of the probability model for all the data considered, not merely that both models fit a selected data set.

For a binomial distribution count, compare a point null $\theta =1/2$ with the uniform alternative prior distribution on $(0,1)$. The alternative Bayesian model evidence isso the exact Bayes factor favouring the point null is $(n+1)\left(\genfrac{}{}{0ex}{}{n}{y}\right)2^{-n}$. With standardized deviation $z=2(y-n/2)/\sqrt{n}$, the local normal distribution approximation is $B_{01}\approx \sqrt{2n/\pi }{e}^{-z^2/2}$. At fixed $z$ the frequentist p-value remains approximately constant, while the factor grows like $\sqrt{n}$. This demonstrates the prior-width mechanism behind the Jeffreys-Lindley paradox for a Gaussian point null in a discrete sampling model.

For a separate original example, change the point null to $H_0:p=1/2$ and take $H_U$ to have a uniform prior $p\sim \mathrm{Beta}(1,1)$, fixed before the ten Bernoulli trials. Here $p$ is drawn once under $H_U$ and shared by all ten trials: they are independent conditional on $p$. For one particular ordered sequence with eight heads and two tails, the Euler beta function givesThus the Bayes factor for the uniform alternative against the point null isThe binomial distribution count formulation gives $m_0(K=8)=45/1024$ and $m_U(K=8)=45/495=1/11$. Their ratio is again $1024/495$; these count masses must not be combined with the ordered-sequence masses above. The general $B_{01}$ formula at the start of this article puts the point null in the numerator, so it equals $B_{0U}$ here.

In contrast, maximum likelihood estimation gives $\hat{p}=0.8$: differentiating the log-likelihood yields $8/p-2/(1-p)=0$, and the endpoints have zero likelihood. Its peak-to-null likelihood ratio isThis peak is chosen after seeing the data, so it is not the Bayes factor of the prespecified $H_U$. The Bayesian model evidence for $H_U$ averages $L(p)$ over its entire uniform prior; it is smaller than the peak. It is also smaller than $L(0.7)=0.7^{8}0.3^2\approx 0.00518832$. Against the same $p=0.5$ null, a point alternative fixed at $p=0.7$ would give a Bayes factor about $5.31284$, greater than $2.06869$. An integrated alternative need not outperform a point alternative.

For a general proper beta distribution prior distribution $\mathrm{Beta}(a,b)$ with $a,b>0$, the same ordered-sequence marginal likelihood is $B(8+a,2+b)/B(a,b)$. This follows by substituting the normalized prior density into the integral, and shows exactly where the parameter prior distribution changes the Bayes factor. Holding the parameter prior distributions fixed while changing model prior odds leaves that Bayes factor unchanged.

A Gaussian practical-null mixture compares a narrow centered normal distribution prior with a wider centered normal distribution prior. Neither component is a point mass. With observed mean $y\mid \beta \sim N(\beta ,1/n)$ and prior precisions $q_0>q_1>0$, its model predictive variances are $V_i=1/n+1/q_i$, and the Bayes factor isThe overall posterior density is a Bayesian model averaging mixture with component means $ny/(n+q_i)$. A wide-prior penalty can favor the practical null near zero; its dominance depends quantitatively on both prior scales and the observation, not merely on the observation being of order $n^{-1/2}$.

Compare an unrestricted model $H_1$ with parameters $(\psi ,\lambda )$ to $H_0:\psi ={\psi }_0$. Suppose the likelihoods agree under that restriction, the priors are proper, and the nuisance prior under $H_0$ equals the conditional prior ${\pi }_1(\lambda \mid {\psi }_0)$ under $H_1$. Using compatible density versions with $0<{\pi }_1({\psi }_0)<\infty$, Bayes' theorem givesIndeed, the numerator of the marginal posterior density at ${\psi }_0$ is ${\pi }_1({\psi }_0)\int L(y\mid {\psi }_0,\lambda ){\pi }_1(\lambda \mid {\psi }_0)d\lambda ={\pi }_1({\psi }_0)m_0(y)$.

This ratio concerns densities in the unrestricted model, not a posterior atom at the null. A separately chosen nuisance prior under the null can invalidate the simple ratio; matching marginal nuisance priors is insufficient when $\psi$ and $\lambda$ are prior-dependent. Density values on a measure-zero restriction also require an agreed continuous or otherwise specified version.^[ref]