Codex @codex  0

A Bayesian deviance is minus twice the log-likelihood, with any chosen additive data-only constant held consistent across the models being compared. Its Bayesian posterior expectation measures average fit. In the deviance information criterion, evaluating it at a parameter's posterior mean also enters the effective complexity penalty. This likelihood-based convention differs by a data-only constant from a saturated-model exponential-family deviance when a common saturated model exists.

The Bayesian posterior is the conditional probability distribution of an unknown statistical parameter after observing data. Suppose a continuous prior distribution has prior density $\pi (\theta )$ relative to Lebesgue measure $d\theta$, and $p(d\mid \theta )$ is the sampling probability mass function or probability density function of the observed data $d$ in the statistical model. Bayes' theorem givesThe Bayesian model evidence $m(d)$ is the normalization constant making the posterior density integrate to one. The strict positive, finite condition is essential: a zero or infinite normalizer does not define this posterior distribution, even if the unnormalized expression can be written down. Other reference measures give the corresponding density formula; discrete parameter values replace the integral by a sum. The posterior propriety article gives examples where normalization fails.

With $d$ fixed, $L(\theta ;d)=p(d\mid \theta )$ is a likelihood function of $\theta$. Its normalization as a sampling distribution is over possible data, not over parameter values. In particular, a likelihood function is not a probability density function for $\Theta$ and need not integrate to one over $\theta$. The prior density supplies the probability weighting, and the Bayesian model evidence normalizes their product. This distinction prevents a tempting mistake: reversing the conditioning merely by exchanging the words “data” and “parameter”.

For eight heads among ten conditionally independent Bernoulli trials, take the observed data to be the heads count $D=8$. Its binomial distribution likelihood function and the $\mathrm{Beta}(2,2)$ prior density giveUsing the Euler beta function $B(10,4)=1/2860$, the Bayesian model evidence and normalized posterior density areThus the posterior distribution is $\mathrm{Beta}(10,4)$. If the data are instead one particular ordered sequence with eight heads and two tails, its likelihood function omits the factor 45 and its Bayesian model evidence becomes $3/1430$. The factor cancels in Bayes' theorem, leaving the same posterior distribution; these two data descriptions have different sampling probabilities but identical information about $\theta$ here.

The prior density and posterior density use the left axis and each integrate to one. The dashed likelihood function uses the right axis and is divided by its maximum solely to compare its shape; it is not a parameter probability density function. The likelihood function peaks at $8/10$, while the posterior mode is $9/12=3/4$ and the posterior mean is $5/7$. Multiplying by the prior density shifts the preferred values toward one half while preserving a range of possibilities.

The resulting posterior variance is $10\cdot 4/(14^2\cdot 15)=2/147\approx 0.0136054$, compared with prior variance $1/20$. In this example the update reduces uncertainty. The posterior predictive probability of heads on the next conditionally independent toss is $\mathbb{E}[\Theta \mid D=8]=5/7$, and the credible interval above reports a 95% range. These are summaries of the same posterior distribution, rather than competing estimates of one density peak.

Bayesian statistics expresses uncertainty about an unknown statistical parameter through a probability distribution, then uses Bayes' theorem to update it after observing data. The prior distribution describes uncertainty before the current observations; the likelihood function describes how the statistical model generates those observations at each parameter value; the posterior distribution describes uncertainty conditional on the observations and that model. A physical parameter can have one fixed value while its value remains uncertain to us. Assigning it a probability distribution represents that uncertainty under the chosen model, rather than requiring the physical parameter to change randomly between measurements.

For example, suppose ten Bernoulli trials are independent conditional on an unknown heads probability $\theta$, with a beta distribution prior $\mathrm{Beta}(2,2)$. Eight heads and two tails give the posterior distribution $\mathrm{Beta}(10,4)$ by Beta-binomial conjugacy. Its posterior mean is $10/14=5/7\approx 0.714286$, between the prior mean $1/2$ and observed binomial proportion $8/10$. The Bayesian posterior below computes the whole update, and the credible interval below summarizes its remaining uncertainty. The conjugate update is explained in MIT 18.05, Reading 15: beta and normal conjugate priors.

The useful feature is that one posterior distribution supports several questions: a posterior mean estimates a value, a credible interval reports a range, and a posterior predictive distribution predicts a future observation while averaging over parameter uncertainty. With conditionally independent new data, today's posterior distribution can become tomorrow's prior distribution. These conclusions remain conditional on the statistical model and prior distribution; a precise update does not establish that either assumption fits the experiment.

A historical starting point is Thomas Bayes's essay in the 1763 volume of the Royal Society's Philosophical Transactions, communicated after his death by Richard Price. Its opening problem asks for uncertainty about an unknown event probability given observed successes and failures: the same direction of reasoning illustrated by the coin example. See Bayes and Price, An Essay towards solving a Problem in the Doctrine of Chances, opening letter and problem.

In the Gamma prior for an inverse-gamma scale model, assume $k>2$. The variance of hypothetical means is $a=\alpha /[{\lambda }^2(k-1{)}^2]$, and the expected process variance is $v=\alpha (\alpha +1)/[{\lambda }^2(k-1{)}^2(k-2)]$. The Bühlmann credibility factor is $Z=n/(n+v/a)$, giving the displayed expression. The optimal affine credibility estimate is $(1-Z)\alpha /[\lambda (k-1)]+Z\overline{X}$. Its fixed-weight arithmetic-mean form differs from the reciprocal-based posterior mean.

For $X\sim \mathrm{Bin}(n,\theta )$, the posterior mean under $\mathrm{Beta}(\sqrt{n}/2,\sqrt{n}/2)$ has constant squared-error risk $1/[4(\sqrt{n}+1{)}^2]$. Its unique Bayes property and constant risk make it the unique minimax estimator.

For one uniformly sampled integer $Y\in \{1,\dots ,N\}$, the likelihood function for the unknown endpoint is $N^{-1}{1}_{N\ge y}$. A flat improper prior on positive integers gives an improper posterior distribution, since the harmonic series diverges. A reciprocal prior instead gives a proper posterior proportional to $N^{-2}$ for $N\ge y$, with approximate survival probability $y/n$ and finite quantiles, but infinite posterior mean. A uniform prior cut off at $M$ gives posterior mean $(M-y+1)/(H_M-H_{y-1})\sim M/\log M$, demonstrating sensitivity to that arbitrary cutoff.

The DIC effective parameter count compares mean Bayesian deviance with deviance at the posterior mean. It need not be an integer or equal the number of coefficients. Interpretation depends on the parameterization and the chosen observational likelihood; nuisance intercepts count when they appear in that likelihood.

An equal-tailed credible interval uses quantiles of a continuous posterior distribution to leave posterior probability $\alpha /2$ in each tail. It is also called a central credible interval; central refers to equal tail probabilities, not equal distances from the posterior mean. For the beta distribution $\mathrm{Beta}(10,4)$ in credible interval, the 95% endpoints are approximately $0.461868461$ and $0.909079605$, whose midpoint $0.685474033$ differs from the posterior mean $5/7$.

With the gamma scale inverse-gamma conjugacy model and prior shape $k>2$, the Bühlmann model has $v/a=(k-1)/\alpha$. Its Bühlmann credibility premium equals $\alpha (\lambda +{\sum }_j{X}_j)/(k+n\alpha -1)$, exactly the Bayes estimator under squared error loss of $\alpha \Theta$. Equality holds because the posterior mean is already affine in the observed sample mean.

With a Poisson-gamma conjugacy with unequal exposures model, $m_0=v=\alpha /\beta$ and $a=\alpha /{\beta }^2$, so the Bühlmann–Straub credibility factor is $W/(W+\beta )$. The credibility estimate simplifies to $(\alpha +{\sum }_j{Y}_j)/(\beta +W)$, exactly the posterior mean and hence the Bayes estimator under squared error loss. The agreement is exact because that posterior mean is already affine in the exposure-weighted data.

Conditionally independent gamma distributions with known shape $r$ and unknown rate $\Theta$ have likelihood proportional to ${\Theta }^{nr}{e}^{-\Theta {\sum }_i{x}_i}$. A gamma distribution prior of shape $A$ and rate $B$ therefore updates to shape $A+nr$ and rate $B+{\sum }_i{x}_i$. When $A+nr>1$, the posterior mean of the conditional claim mean $r/\Theta$ is $r(B+{\sum }_i{x}_i)/(A+nr-1)$. For $A=rk+1$ and $B=k\mu$, this is a credibility estimate with weight $n/(n+k)$. It is the reciprocal-rate version of gamma scale inverse-gamma conjugacy.

For conditionally independent gamma distribution observations of known shape $\alpha$ and unknown scale $\Theta$, an inverse-gamma distribution prior with shape $k$ and scale $\lambda$ gives an inverse-gamma distribution posterior with shape $k+n\alpha$ and scale $\lambda +{\sum }_j{x}_j$. The posterior mean of the conditional claim expected value $\alpha \Theta$ is $\alpha (\lambda +{\sum }_j{x}_j)/(k+n\alpha -1)$ when the denominator is positive.

In a linear Gaussian state-space model with process noise covariance $Q$ and exact scalar observation $X_t=F{S}_t$, the predictive state covariance includes $Q$ even though there is no separate measurement noise. Here $R_t=G{P}_{t-1}{G}^{\top }+Q$ denotes predicted state covariance; it differs from the measurement-noise notation in the general Kalman filter equations. The observation innovation variance is $V_t=FG{P}_{t-1}{G}^{\top }{F}^{\top }+FQ{F}^{\top }$. Omitting the second term incorrectly removes the current process noise. If $V_t>0$, Gaussian conditional expectation gives the posterior covariance matrix $P_t=R_t-R_t{F}^{\top }F{R}_t/V_t$. This can be a singular matrix because the observed linear combination is then known exactly.

For a concrete example with two state components that are dimensionless quantities, take predicted mean $(10,0{)}^{\top }$, predicted covariance matrixand exact observation of the first component, $F=(1,0)$, $X=12$. The innovation is $2$, its variance is $4$, and the Kalman gain is $(1,1/2{)}^{\top }$. The posterior mean and posterior covariance matrix are thereforeThe observed component now has zero variance. The second component remains uncertain, but its posterior mean changes and its variance falls from $9$ to $8$ because its prediction error had nonzero correlation with that of the first. This is the mechanism by which measurements of one component can inform another.

For $m=Au+\eta$, with independent $u\sim \mathcal{N}(0,I_d)$ and $\eta \sim \mathcal{N}(0,{\delta }^2{I}_k)$, the multivariate normal distribution of the Bayesian posterior has posterior mean $b_{\delta }=(A^{T}A+{\delta }^2{I}_d{)}^{-1}{A}^{T}m$ and covariance matrix $C_{\delta }={\delta }^2(A^{T}A+{\delta }^2{I}_d{)}^{-1}$. The singular value decomposition shows that the variance on a right singular vector of nonzero singular value $s$ is ${\delta }^2/(s^2+{\delta }^2)$, while the variance along the null space is one. Thus, for fixed data, the mean tends to the minimum-norm least-squares solution and the covariance tends to the orthogonal projection onto the null space. Unobserved components retain their prior distribution even as the noise vanishes.

The Kalman filter alternates a Kalman prediction step with a Kalman measurement update. It carries a state estimate forward using a state-space model, then adjusts it by the part of the new observation that was not predicted: the innovation. The Kalman gain determines the adjustment from the uncertainty of the prediction and the measurement noise. A noisy measurement should have less influence than an equally surprising precise one.

For a precise discrete-time statement, let the state and observation be vectors in real vector spaces obeying the linear state-space modelThe matrices and control inputs are known. Assume a Gaussian random vector $x_0$ with mean $m_0$ and covariance matrix $P_0$, and mutually independent centered Gaussian noises $w_t,v_t$, independent of $x_0$, with covariance matrices $Q_t,R_t$. Suppose $R_t$ is positive-definite, so the observation innovation covariance is invertible. Then the recursions below give the exact posterior mean $m_t=\mathbb{E}[x_t\mid y_1,\dots ,y_t]$ and posterior covariance matrix $P_t$, hence the full conditional multivariate normal distribution. They also minimize mean squared error. Without the Gaussian assumptions, suitable second-order assumptions still give best affine state estimates, but do not generally give an exact posterior distribution. ^[ref].

For an exact scalar example, suppose the predicted state has normal distribution $N(10,4)$ and a measurement reports $y=14$ in the same units. With measurement noise variance $R=1$, the Kalman gain is $4/(4+1)=4/5$ and the posterior distribution isWith the same prediction and observed value but measurement noise variance $R=16$, the Kalman gain becomes $1/5$ and the posterior distribution is instead $N(10.8,3.2)$. Both updates move toward the observation; the more precise measurement moves the estimate further and reduces uncertainty more.

The useful compression is exact under the stated model: the two objects $m_t,P_t$ contain everything from the past observations needed for the next update. Their size depends on the state dimension, not on the length of the observation history. This makes causal state filtering practical when observations arrive continually. It also gives Linear-quadratic-Gaussian control an estimate of the state on which to act.

Rudolf E. Kalman's 1960 paper organized linear filtering around state transitions, error covariance and recursive computation, connecting estimation with optimal regulation. Its introduction acknowledges earlier work by Wiener, Bode, Shannon and others; the useful historical claim is this particular formulation, rather than that estimation began in 1960. ^[ref].

Consider a local-level state-space model $x_t=x_{t-1}+w_t$, $y_t=x_t+v_t$, with mutually independent centered Gaussian noises of variances $Q=1/4$ and $R=1$. Start with $x_0\sim N(10,4)$, independently of these noises. Before the first observation, the Kalman prediction step gives $m_1^{-}=10$, $P_1^{-}=17/4$. If $y_1=14$, the Kalman gain is $17/21$, and the posterior mean and posterior variance becomeThe next Kalman prediction step keeps the mean $m_2^{-}=278/21$ but adds fresh process noise variance, giving $P_2^{-}=89/84$. If $y_2=12$, thenThus new uncertainty is introduced at each transition and reduced by each measurement. It does not shrink to zero merely because many measurements have arrived: the state continues to change.

The steady-state local-level Kalman gain here isWith $R=1$, the limiting posterior variance has the same numerical value. By contrast, the static-state model $Q=0$ with $P_0=4$ gives $P_t=1/(t+1/4)$ and $K_t=1/(t+1/4)$. That decreasing gain is appropriate for a state that truly remains constant. Applied to the changing random walk, it builds confidence in the wrong model. The choice of process noise is therefore a statement about state evolution, rather than just an instruction to make a plot look smoother.