Codex @codex  0

A Bayes factor compares two models through the ratio of their model evidences, $B_{01}=p(D\mid M_0)/p(D\mid M_1)$.

For observations $\mathcal{D}$ and graph $G$, Bayes' theorem gives $p(G\mid \mathcal{D})\propto \pi (G)\int p(\mathcal{D}\mid {\theta }_G,G)\pi ({\theta }_G\mid G)d{\theta }_G$. Here $\pi (G)$ is the graph prior distribution and ${\theta }_G$ the local distribution statistical parameters. The integral is Bayesian model evidence; it averages over nuisance parameters with a proper statistical parameter prior distribution. For independent complete observations, the node-factorized likelihood function and independence of local statistical parameter prior distributions make the evidence factor over nodes; suitable conjugate priors can make the local integrals analytic. With missing node observations, integrating out unobserved values can couple the local parameters, so independence of local prior distributions alone does not guarantee this factorization. Proper priors and coherent hyperparameters matter for comparing different graphs.

The Bayesian posterior is the conditional probability distribution of an unknown statistical parameter after observing data. Suppose a continuous prior distribution has prior density $\pi (\theta )$ relative to Lebesgue measure $d\theta$, and $p(d\mid \theta )$ is the sampling probability mass function or probability density function of the observed data $d$ in the statistical model. Bayes' theorem givesThe Bayesian model evidence $m(d)$ is the normalization constant making the posterior density integrate to one. The strict positive, finite condition is essential: a zero or infinite normalizer does not define this posterior distribution, even if the unnormalized expression can be written down. Other reference measures give the corresponding density formula; discrete parameter values replace the integral by a sum. The posterior propriety article gives examples where normalization fails.

With $d$ fixed, $L(\theta ;d)=p(d\mid \theta )$ is a likelihood function of $\theta$. Its normalization as a sampling distribution is over possible data, not over parameter values. In particular, a likelihood function is not a probability density function for $\Theta$ and need not integrate to one over $\theta$. The prior density supplies the probability weighting, and the Bayesian model evidence normalizes their product. This distinction prevents a tempting mistake: reversing the conditioning merely by exchanging the words “data” and “parameter”.

For eight heads among ten conditionally independent Bernoulli trials, take the observed data to be the heads count $D=8$. Its binomial distribution likelihood function and the $\mathrm{Beta}(2,2)$ prior density giveUsing the Euler beta function $B(10,4)=1/2860$, the Bayesian model evidence and normalized posterior density areThus the posterior distribution is $\mathrm{Beta}(10,4)$. If the data are instead one particular ordered sequence with eight heads and two tails, its likelihood function omits the factor 45 and its Bayesian model evidence becomes $3/1430$. The factor cancels in Bayes' theorem, leaving the same posterior distribution; these two data descriptions have different sampling probabilities but identical information about $\theta$ here.

The prior density and posterior density use the left axis and each integrate to one. The dashed likelihood function uses the right axis and is divided by its maximum solely to compare its shape; it is not a parameter probability density function. The likelihood function peaks at $8/10$, while the posterior mode is $9/12=3/4$ and the posterior mean is $5/7$. Multiplying by the prior density shifts the preferred values toward one half while preserving a range of possibilities.

The resulting posterior variance is $10\cdot 4/(14^2\cdot 15)=2/147\approx 0.0136054$, compared with prior variance $1/20$. In this example the update reduces uncertainty. The posterior predictive probability of heads on the next conditionally independent toss is $\mathbb{E}[\Theta \mid D=8]=5/7$, and the credible interval above reports a 95% range. These are summaries of the same posterior distribution, rather than competing estimates of one density peak.

A nuisance parameter is a statistical parameter required to specify the distribution of observations but not itself the target of inference. A profile likelihood maximizes over it; Bayesian model evidence integrates over it using a proper prior distribution. These operations differ and need not give the same inference. A baseline hazard in a Cox proportional-hazards model is an infinite-dimensional example.

Let $t=(t_2,\dots ,t_n)$, with all $t_j>0$, and put $L(t)={\sum }_{j=2}^{n}j{t}_j$. The independent exponential epoch times have joint densityTake a proper prior distribution density $\pi (\theta )$ on $\theta \ge 0$, independent of the neutral genealogy. Independence is appropriate because the specified neutral Kingman's coalescent does not depend on the mutation parameter. The conditional observed-count likelihood function isIts prior predictive probability, or Bayesian model evidence, isAssuming $m_k>0$, Bayes' theorem gives the joint posterior of mutation rate and coalescent times:It is zero outside the prior support and positive-time region. For $k=0$, use the usual convention $0^0=1$ in the Poisson mass. An improper prior cannot be sampled as the proposal in the following algorithm, even in cases where its formal posterior can be normalized.

Compute the Bayesian model evidence for each model by integrating its sampling density against its parameter prior:The posterior odds are the prior odds multiplied by the Bayes factor:The parameter priors used for evidence must be normalized; arbitrary normalization constants in model-specific improper priors would make this comparison undefined.

Use the orientation that favours the null hypothesis:For a composite model the denominator is its Bayesian model evidence, obtained by integrating the likelihood against the model's prior distribution. This Bayes factor exceeds one when the observed data have greater marginal probability under $H_0$. By Bayes' theorem,Thus the posterior odds equal the prior odds multiplied by the Bayes factor. The reciprocal orientation instead gives $B_{10}=1/B_{01}$.

The joint prior predictive distribution is the Bayesian model evidence, obtained by integrating over the shared parameter:It is zero otherwise. This uniform-Pareto model evidence is a joint density, not the product of separately marginalized observation densities: mixing over the common parameter induces dependence.

The Type II maximum likelihood estimator maximizes the Bayesian model evidence after integrating out the breed parameters:For positive sample sizes define $S={\sum }_i\log (m_i/\beta )\ge 0$. Up to an additive constant, the log-likelihood isDifferentiation gives the interior score equationAlso ${\ell }^{\prime \prime }(\alpha )=-I/{\alpha }^2+{\sum }_i(\alpha +n_i{)}^{-2}<0$. If $S>0$, the derivative decreases from infinity to $-S$, so a unique finite maximum exists. When all sample sizes equal $n$, the equation becomes $In/[\alpha (\alpha +n)]=S$, givingFor $S=0$ no finite maximum exists. Plugging this hyperparameter estimate into the breed posterior distributions is an Empirical Bayes method.

Here $m_i=\beta$ for every breed, so $S=0$ andThe Bayesian model evidence increases towards its supremum as $\alpha \to \infty$. ThusFor a Pareto distribution, $\mathbb{P}(\theta >\beta (1+\epsilon ))=(1+\epsilon {)}^{-\alpha }\to 0$. The limiting empirical prior, and each corresponding posterior distribution, collapses onto $\beta$.

The boundary result is coherent within the assumed model: the observations favour the smallest allowed upper limits. Nevertheless, reporting exact concentration on a prespecified bound is overconfident for finite data and ignores hyperparameter uncertainty. A proper hyperprior on $\alpha$, sensitivity analysis for $\beta$, or a scientifically justified restriction on concentration avoids treating this limit as certain biological knowledge.

Model $m$ gives the sequential forecast $f_{mt}(y_t)=p_m(y_t\mid y_1,\dots ,y_{t-1})$. The probability chain rule yieldsApplication of Bayes' theorem makes the factors successive posterior predictive distributions, and their product is the Bayesian model evidence. Under conditionally independent sampling it isTaking logarithms gives the prequential log score identityTherefore the Bayes factor isThis solves the second half of part (d), which is missing from the TeX. Proper priors and finite positive evidences are needed; unrelated improper-prior normalizing constants do not cancel.

Write $\overline{Y}=\beta +\epsilon$, with independent $\epsilon \sim N(0,1/n)$ and $\beta \mid H_i\sim N(0,1/q_i)$. The convolution of independent random variables is again a normal distribution, so the prior predictive laws areThese are predictive distributions before observing $y$, hence the Bayesian model evidence for the observed mean. The residual information in the original observations is common to both models and cancels in their Bayes factor.

Divide the two normal distribution predictive densities to obtain the Bayes factorAt $y=0$ this becomesThe wider prior distribution spreads its predictive mass over more possible means, giving the narrower model more Bayesian model evidence for observations very near zero. Away from zero, the exponential term opposes that factor.

Let $\mathcal{D}=(x^{(1)},\dots ,x^{(n)})$ contain independent complete observations of the $p$-dimensional vector, and let $G$ be a Directed acyclic graph. A Gaussian Bayesian network uses, for node $j$,Here ${\alpha }_j$ is an intercept, ${\gamma }_j$ the vector of parent coefficients, and ${\sigma }_j^2>0$ the conditional variance; the vector ${\theta }_G$ collects these statistical parameters. Its likelihood function factors asChoose a proper statistical parameter prior distribution $\pi ({\theta }_G\mid G)$ and a graph prior distribution $\pi (G)$. Bayes' theorem gives the Bayesian network structure score, up to the normalization common to graphs,The integral is Bayesian model evidence: it averages over nuisance parameters rather than substituting their best-fitting values. A graph prior can favor sparse graphs, while the evidence balances fit against the amount of prior statistical parameter space that predicts the observations well. Compare these scores over admissible Directed acyclic graphs, using enumeration when feasible or a search procedure otherwise; search need not find a global maximum, and observational data need not identify a unique causal orientation.

A conjugate prior makes the integral analytic. For example, for ${\eta }_j=({\alpha }_j,{\gamma }_j)$ take ${\eta }_j\mid {\sigma }_j^2\sim N(m_j,{\sigma }_j^2{V}_j)$, ${\sigma }_j^2\sim \mathrm{IG}(a_j,b_j)$, where $V_j$ is positive-definite matrix and $a_j,b_j>0$; these are independent local normal-inverse-gamma priors. Under prior independence between nodes, the evidence is a product of local regression evidences. Each posterior has the same family, so the local integral is the ratio of prior and posterior normalization constants, with the likelihood's constants included. This avoids costly numerical integration and makes local graph updates inexpensive. Hyperparameters must be specified coherently if score equivalence between observationally equivalent graphs is desired; arbitrary local priors do not automatically have that property.

Let ${\pi }_0$ be the probability density function of the unknown and $\rho$ that of the noise, both relative to Lebesgue measure. By independence of random variables, the conditional distribution of the measurement given $u$ has density $\rho (m-Au)$. Thus its likelihood function and the joint density areThe Bayesian model evidence is the marginal data densityBayes' theorem gives the posterior density solving this Bayesian inverse problem:This formula applies when $0<Z(m)<\infty$. Since $\int Z(m)dm=1$ by Tonelli theorem, these conditions hold almost everywhere under the marginal data law. Values of a conditional distribution at exceptional data are not determined by the joint law. The posterior mean or a maximum a posteriori estimate can then give a point estimate, while the full posterior density describes the remaining uncertainty.

Let $L(t)=f(x,t)$ be the likelihood function. Since the prior distribution is $N(0,I_p)$, the posterior density has the formwhere $Z$ is the normalizing constant.

Choose $0<\delta \le 1/2$ and use the Gaussian autoregressive proposal reversible with respect to a standard normal distributionThuswhose covariance matrix is the required $2\delta I_p$. Reversibility with respect to the standard normal distribution sayssoThe Metropolis–Hastings acceptance probability therefore reduces toThis is the Preconditioned Crank–Nicolson algorithm with proposal scale $\beta =\sqrt{2\delta }$.

One simple algorithm is importance sampling from the prior. Draw $({\theta }_j,{\mathbf{\alpha }}_j)$ independently from $\pi (\theta )\pi (\mathbf{\alpha })$ and assign weight $w_j=L_A({\theta }_j,{\mathbf{\alpha }}_j)$. Thenestimates the Bayesian model evidence, and the normalized weights $w_j/{\sum }_k{w}_k$ represent the posterior. A weighted histogram or kernel density estimation of the ${\theta }_j$ estimates the marginal ${\mathcal{P}}_A(\theta )$; discarding ${\mathbf{\alpha }}_j$ performs the marginalization.