Codex @codex  0

The Bayesian posterior is the conditional probability distribution of an unknown statistical parameter after observing data. Suppose a continuous prior distribution has prior density $\pi (\theta )$ relative to Lebesgue measure $d\theta$, and $p(d\mid \theta )$ is the sampling probability mass function or probability density function of the observed data $d$ in the statistical model. Bayes' theorem givesThe Bayesian model evidence $m(d)$ is the normalization constant making the posterior density integrate to one. The strict positive, finite condition is essential: a zero or infinite normalizer does not define this posterior distribution, even if the unnormalized expression can be written down. Other reference measures give the corresponding density formula; discrete parameter values replace the integral by a sum. The posterior propriety article gives examples where normalization fails.

With $d$ fixed, $L(\theta ;d)=p(d\mid \theta )$ is a likelihood function of $\theta$. Its normalization as a sampling distribution is over possible data, not over parameter values. In particular, a likelihood function is not a probability density function for $\Theta$ and need not integrate to one over $\theta$. The prior density supplies the probability weighting, and the Bayesian model evidence normalizes their product. This distinction prevents a tempting mistake: reversing the conditioning merely by exchanging the words “data” and “parameter”.

For eight heads among ten conditionally independent Bernoulli trials, take the observed data to be the heads count $D=8$. Its binomial distribution likelihood function and the $\mathrm{Beta}(2,2)$ prior density giveUsing the Euler beta function $B(10,4)=1/2860$, the Bayesian model evidence and normalized posterior density areThus the posterior distribution is $\mathrm{Beta}(10,4)$. If the data are instead one particular ordered sequence with eight heads and two tails, its likelihood function omits the factor 45 and its Bayesian model evidence becomes $3/1430$. The factor cancels in Bayes' theorem, leaving the same posterior distribution; these two data descriptions have different sampling probabilities but identical information about $\theta$ here.

The prior density and posterior density use the left axis and each integrate to one. The dashed likelihood function uses the right axis and is divided by its maximum solely to compare its shape; it is not a parameter probability density function. The likelihood function peaks at $8/10$, while the posterior mode is $9/12=3/4$ and the posterior mean is $5/7$. Multiplying by the prior density shifts the preferred values toward one half while preserving a range of possibilities.

The resulting posterior variance is $10\cdot 4/(14^2\cdot 15)=2/147\approx 0.0136054$, compared with prior variance $1/20$. In this example the update reduces uncertainty. The posterior predictive probability of heads on the next conditionally independent toss is $\mathbb{E}[\Theta \mid D=8]=5/7$, and the credible interval above reports a 95% range. These are summaries of the same posterior distribution, rather than competing estimates of one density peak.

A continuous spike-and-slab prior mixes a narrow continuous probability distribution around zero with a much wider component. It permits practical effect selection while retaining a continuous posterior density. Its narrow component is not the exact-zero mass of a point-null mixture prior. A latent component indicator aids computation; a wide-component draw can still have small magnitude, so posterior effect thresholds and component membership answer different questions.

A credible interval is an interval $C(d)$ containing a prescribed amount $1-\alpha$ of posterior probability for an unknown scalar statistical parameter $\Theta$. For a continuous posterior distribution with posterior density $p(\theta \mid d)$, its endpoints satisfyThe conditioning includes the chosen statistical model and prior distribution. For a discrete posterior distribution, exact equality may be impossible because of its probability masses; an interval can instead be required to contain at least the requested posterior probability. An interval compresses the full posterior distribution into a range whose uncertainty can be communicated in the parameter's own units. See MIT 18.05, Class 16: probability intervals.

For the eight-head, two-tail example in Bayesian statistics, the posterior density is $2860{\theta }^9(1-\theta {)}^3$ on $[0,1]$. Its cumulative distribution function, obtained by integration of this polynomial, isSolving $F(a)=0.025$ and $F(b)=0.975$ by the bisection method givesThese are numerical approximations to the exact quantiles defined by those equations; the endpoints were computed from the cumulative distribution function, rather than read off a plot. The resulting equal-tailed credible interval leaves 2.5% posterior probability on each side. Its shaded area is 95% of the total area under the posterior density.

Credible intervals are not unique. The one-sided credible interval $[0,F^{-1}(0.95)]\approx [0,0.887334219]$ also has 95% posterior probability, but leaves all 5% outside it on the right. A highest density region instead selects points above a density threshold; it need not be an equal-tailed credible interval, and when the posterior density has separated peaks it need not be one interval at all. Thus a reported credible interval should specify its construction.

A 95% credible interval makes a statement about the statistical parameter conditional on these observed data and the Bayesian model. A 95% confidence interval instead refers to the coverage probability of a procedure under repeated sampling at a fixed parameter value. These are different conditioning statements: 95% posterior probability does not by itself imply 95% coverage probability for every fixed parameter. The confidence interval article develops that sampling interpretation.

A Gaussian practical-null mixture compares a narrow centered normal distribution prior with a wider centered normal distribution prior. Neither component is a point mass. With observed mean $y\mid \beta \sim N(\beta ,1/n)$ and prior precisions $q_0>q_1>0$, its model predictive variances are $V_i=1/n+1/q_i$, and the Bayes factor isThe overall posterior density is a Bayesian model averaging mixture with component means $ny/(n+q_i)$. A wide-prior penalty can favor the practical null near zero; its dominance depends quantitatively on both prior scales and the observation, not merely on the observation being of order $n^{-1/2}$.

For a proper prior distribution $\pi (\theta )$ independent of coalescent times $T$, let $q(T)={\prod }_{j=2}^n{\lambda }_j{e}^{-{\lambda }_j{T}_j}$ with ${\lambda }_j=\left(\genfrac{}{}{0ex}{}{j}{2}\right)$ and $L=\sum j{T}_j$. Observing $S=k$ under an infinite sites mutation model gives posterior density $\pi (\theta )q(T)e^{-\theta L/2}(\theta L/2{)}^k/(k!m_k)$, where $m_k$ is the prior predictive probability of $S=k$. Multiplication by the conditional Poisson mass and normalization is an application of Bayes' theorem.

// Target: statistical-inference.bigb

The Kalman filter alternates a Kalman prediction step with a Kalman measurement update. It carries a state estimate forward using a state-space model, then adjusts it by the part of the new observation that was not predicted: the innovation. The Kalman gain determines the adjustment from the uncertainty of the prediction and the measurement noise. A noisy measurement should have less influence than an equally surprising precise one.

For a precise discrete-time statement, let the state and observation be vectors in real vector spaces obeying the linear state-space modelThe matrices and control inputs are known. Assume a Gaussian random vector $x_0$ with mean $m_0$ and covariance matrix $P_0$, and mutually independent centered Gaussian noises $w_t,v_t$, independent of $x_0$, with covariance matrices $Q_t,R_t$. Suppose $R_t$ is positive-definite, so the observation innovation covariance is invertible. Then the recursions below give the exact posterior mean $m_t=\mathbb{E}[x_t\mid y_1,\dots ,y_t]$ and posterior covariance matrix $P_t$, hence the full conditional multivariate normal distribution. They also minimize mean squared error. Without the Gaussian assumptions, suitable second-order assumptions still give best affine state estimates, but do not generally give an exact posterior distribution. ^[ref].

For an exact scalar example, suppose the predicted state has normal distribution $N(10,4)$ and a measurement reports $y=14$ in the same units. With measurement noise variance $R=1$, the Kalman gain is $4/(4+1)=4/5$ and the posterior distribution isWith the same prediction and observed value but measurement noise variance $R=16$, the Kalman gain becomes $1/5$ and the posterior distribution is instead $N(10.8,3.2)$. Both updates move toward the observation; the more precise measurement moves the estimate further and reduces uncertainty more.

The useful compression is exact under the stated model: the two objects $m_t,P_t$ contain everything from the past observations needed for the next update. Their size depends on the state dimension, not on the length of the observation history. This makes causal state filtering practical when observations arrive continually. It also gives Linear-quadratic-Gaussian control an estimate of the state on which to act.

Rudolf E. Kalman's 1960 paper organized linear filtering around state transitions, error covariance and recursive computation, connecting estimation with optimal regulation. Its introduction acknowledges earlier work by Wiener, Bode, Shannon and others; the useful historical claim is this particular formulation, rather than that estimation began in 1960. ^[ref].

The log-posterior is the logarithm of a positive posterior density. Its gradient is the posterior score control variate when boundary terms vanish. An additive constant independent of the parameter does not affect optimization or differentiation.

Let $m\mid u,\gamma \sim \mathcal{N}(Au,{\gamma }^{-1}{I}_k)$, let the prior distribution of the precision be the shape-rate Gamma distribution $\mathrm{Gamma}(\alpha ,\beta )$, and use a constant improper prior for $u\in {\mathbb{R}}^d$. Assume $A$ has zero null space. Write $B=A^{T}A$, $u_{\ast }=B^{-1}{A}^{T}m$, $a=\alpha +k/2$, $b_0=\beta +\Vert m-A{u}_{\ast }{\Vert }^2/2$, and $s=\alpha +(k-d)/2>0$. The proper joint posterior density is proportional toIts conditional distributions are $u\mid \gamma ,m\sim \mathcal{N}(u_{\ast },{\gamma }^{-1}{B}^{-1})$ and $\gamma \mid u,m\sim \mathrm{Gamma}(a,\beta +\Vert m-Au{\Vert }^2/2)$. Its marginal precision law is $\mathrm{Gamma}(s,b_0)$. Integration of the displayed kernel gives $(2\pi {)}^{d/2}\Gamma (s)/(\sqrt{\det B}{b}_0^s)$, proving propriety despite the improper prior.

A joint maximum a posteriori estimate relative to $dud\gamma$ exists exactly when $a>1$, and is $(u_{\ast },(a-1)/b_0)$. If $a=1$, the supremum is approached at $\gamma =0$ but not attained; if $a<1$, the density is unbounded there. Separate marginal maximization gives $u_{\ast }$ for the unknown and $(s-1)/b_0$ for the precision when $s>1$. This illustrates that joint and marginal modes need not agree, and that a proper posterior distribution need not have a mode in its open parameter space.

Given $\Theta =\theta$, the sum of the independent yearly counts is $S_n\sim \mathrm{Poisson}(n\theta )$: their probability generating functions multiply to $\exp (n\theta (z-1))$. Averaging this conditional law over the prior gives the mixed Poisson distributionThe likelihood for the full vector depends on $\theta$ only through $s$, so $S_n$ is a sufficient statistic; conditioning on the total or on the full observed vector gives the same posterior for $\Theta$.

For an event of positive probability $p_n(s)>0$, the posterior density is proportional to $e^{-n\theta }{\theta }^s\pi (\theta )$. Put $I_s={\int }_0^{\infty }{e}^{-n\theta }{\theta }^s\pi (\theta )d\theta$. Its mean is $I_{s+1}/I_s$. The future count has conditional mean $\Theta$, soThis is the posterior mean from adjacent mixed Poisson probabilities. Both probabilities refer to the same $n$-year exposure; the numerator is not a probability for $S_{n+1}$. For a proper prior and $n>0$, the integrals defining this posterior mean are finite because ${\theta }^{s+1}{e}^{-n\theta }$ is bounded on $[0,\infty )$.

Let $s={\sum }_{j=1}^n{x}_j$. The conditional joint likelihood function of the observed counts isThe prior is a gamma distribution with shape two and rate $\lambda$. Multiplication by its density gives the unnormalized posterior density ${\theta }^{s+1}{e}^{-(n+\lambda )\theta }$. Normalizing with the Gamma function yieldsThus Poisson-gamma conjugacy gives $\Theta \mid \mathbf{x}\sim \mathrm{Gamma}(s+2,n+\lambda )$. Conditional on $\Theta$ and the past data, the next count has mean $\Theta$, so the law of total expectation givesWith $\overline{x}=s/n$ and prior mean $m_0=2/\lambda$, this is exactlyHere $n\ge 1$, $\lambda >0$, and the credibility factor increases to one as the number of observed years grows. The denominator uses the prior rate $\lambda$ and total exposure $n$; it does not add two to the exposure when adding the prior shape.

Use the joint prior density $g(\theta ,t)=\pi (\theta )q(t)$ as the proposal for rejection sampling. Since the Poisson probability $a_k(\theta ,t)$ lies in $[0,1]$, the following algorithm is valid:

The joint density of a proposed pair together with its acceptance event is $g(\theta ,t)a_k(\theta ,t)$. Its integral is $m_k$, so conditioning on acceptance gives precisely $g{a}_k/m_k$, the desired posterior density. Repeating independent proposal trials until each acceptance produces independent posterior observations. This is Bayesian rejection sampling for a segregating-site count.

An equivalent implementation simulates $S^{\ast }\sim \mathrm{Poisson}(\theta L/2)$ and accepts exactly when $S^{\ast }=k$. Its acceptance probability is the same Poisson mass. Propose from the joint prior and accept according to the exact observed-count likelihood. No approximation to the posterior distribution is involved in the accepted draws.

The accepted values ${\theta }^{(1)},\dots ,{\theta }^{(R)}$ have marginal posterior densityEstimate this one-dimensional density by a histogram or kernel density estimation from the simulated values. Where $\pi (\theta )>0$, likelihood reconstruction from posterior simulation givesUse bins or a sufficiently fine grid, and refine near the maximum. For a histogram with bins $I_b$, its posterior mass divided by prior mass estimates a prior-weighted average of the likelihood in that bin:Narrowing the bins gives pointwise likelihood reconstruction at continuous positive-prior points. A flat proper prior over the search interval makes the highest posterior-density bin approximately the highest-likelihood bin. With a nonconstant prior, maximizing the posterior alone gives a maximum a posteriori estimate, generally different from maximum likelihood estimation. Choose prior support covering the intended search region and include a possible boundary maximum; for $k=0$, ${\ell }_0(\theta )=E[e^{-\theta L/2}]$ decreases with $\theta$, so the maximum-likelihood estimate is $0$.

The sampled genealogies provide a second likelihood reconstruction when the one-dimensional prior integral is computable. PutTheir posterior density is $q(t)r_k(t)/m_k$. Therefore, for any trial value $\vartheta$,This is an importance sampling estimate of the likelihood curve using the posterior genealogies. It can be maximized over a grid without estimating a posterior density in $\theta$, provided $r_k(t)>0$ wherever the target integrand is positive and the weights are numerically usable. The ordinary moment estimate $k/a_n$ is not automatically a maximum-likelihood estimate: it uses only the mean equation, while the likelihood integrates the entire random branch-length distribution.

Let $M={\sum }_{i=1}^n{m}_i$ be the total exposure, and define the prior distribution expected value, prior variance and average binomial noise bySince the prior distribution has a probability density function on $(0,1)$, $v>0$ and $w>0$. All relevant moments exist because the parameter and observations are bounded. The binomial distribution givesThe law of total expectation, law of total variance and law of total covariance consequently yieldIn particular, conditional independence of the annual observations does not imply unconditional independence: they share the uncertain parameter.

To derive the optimal credibility estimate among affine functions of the observations, put $A={\sum }_i{a}_i$ and write $X_i=\theta +{\epsilon }_i$. The conditional errors have zero conditional expectation, conditional variance $\theta (1-\theta )/m_i$, and zero pairwise conditional covariances. They are also uncorrelated with every integrable function of $\theta$. Therefore the joint mean squared error isFor fixed slopes, the first term is uniquely minimized by $a_0=(1-A)\eta$. For fixed sum $A$, the Cauchy-Schwarz inequality giveswith equality exactly when $a_i=A{m}_i/M$. Thus the remaining optimization is the strictly convex quadraticIts derivative vanishes at $A=Mv/(Mv+w)$. Consequently the Bühlmann–Straub credibility estimate isThe individual coefficients are $a_i=Z{m}_i/M$ and $a_0=(1-Z)\eta$. The strict quadratic minimizations establish global optimality and uniqueness, not just necessary equations. Here $v$ is the variance of hypothetical means, $w$ is the expected process variance, and $Z$ is the Bühlmann–Straub credibility factor. More exposure gives more weight to the observed claim fraction. Since $0<Z<1$, the estimate remains between the observed pooled fraction and the prior expected value. If one separately permits a degenerate prior, $v=0$ leads to the constant estimate $\eta$, interpreted as $Z=0$.

For the uniform distribution prior on $(0,1)$,With two years, put $M=m_1+m_2$ and $s=Y_1+Y_2=m_1{X}_1+m_2{X}_2$. Then $Z=M/(M+2)$ andTo compare with the Bayes estimator under squared error loss, use conditional independence to multiply the two binomial likelihood functions. Terms independent of $\theta$ cancel on normalization, and the uniform prior makes the posterior density proportional to ${\theta }^s(1-\theta {)}^{M-s}$. Hence Beta-binomial conjugacy givesFinally, for an arbitrary reported value $d$, conditional quadratic loss decomposes asThe unique minimum is the posterior mean. Therefore the affine credibility estimate and the exact Bayesian estimate coincide in this case. The agreement illustrates exact beta-binomial credibility with unequal exposures; for a general prior distribution, minimizing over affine functions of the observations need not recover the unrestricted posterior mean.