Conjugate prior

A conjugate prior belongs to a family whose posterior remains in that family after multiplication by the likelihood.

A normal-gamma conjugate prior for the mean $\mu$ and precision parameter $\tau$ of a Gaussian distribution specifies a Gamma distribution $\tau \sim \mathrm{Gamma}(\alpha ,\beta )$ in the shape-rate convention and a conditional Gaussian distribution $\mu \mid \tau \sim N({\mu }_0,(K\tau {)}^{-1})$. Its joint probability density function is proportional to ${\tau }^{\alpha -1}\sqrt{\tau }\exp [-\beta \tau -K\tau (\mu -{\mu }_0{)}^2/2]$, with $\alpha ,\beta ,K>0$. For $n$ independent observations with Gaussian distribution $N(\mu ,{\tau }^{-1})$, multiplication by the likelihood and completing the square givesand ${\beta }_n=\beta +{\sum }_i(x_i-\bar{x}{)}^2/2+Kn(\bar{x}-{\mu }_0{)}^2/[2(K+n)]$. The identity behind conjugacy is $K(\mu -{\mu }_0{)}^2+{\sum }_i(x_i-\mu {)}^2=(K+n)(\mu -{\mu }_n{)}^2+{\sum }_i(x_i-\bar{x}{)}^2+Kn(\bar{x}-{\mu }_0{)}^2/(K+n)$.

Conditionally independent gamma distributions with known shape $r$ and unknown rate $\Theta$ have likelihood proportional to ${\Theta }^{nr}{e}^{-\Theta {\sum }_i{x}_i}$. A gamma distribution prior of shape $A$ and rate $B$ therefore updates to shape $A+nr$ and rate $B+{\sum }_i{x}_i$. When $A+nr>1$, the posterior mean of the conditional claim mean $r/\Theta$ is $r(B+{\sum }_i{x}_i)/(A+nr-1)$. For $A=rk+1$ and $B=k\mu$, this is a credibility estimate with weight $n/(n+k)$. It is the reciprocal-rate version of gamma scale inverse-gamma conjugacy.

A Pareto distribution prior for the upper limit of independent uniform distributions remains Pareto after observing data. The likelihood adds the sample size to the shape and replaces the lower bound by the larger of the prior bound and sample maximum.

For positive observations and $m=\max (\beta ,{\max }_j{y}_j)$, the Bayesian model evidence follows by integrating the uniform distribution likelihood against a Pareto distribution prior. For independent groups with shared fixed hyperparameters, these evidences multiply. The resulting hyperparameter log likelihood is $I\log \alpha -{\sum }_i\log (\alpha +n_i)-\alpha {\sum }_i\log (m_i/\beta )$ up to a constant.

For conditionally independent gamma distribution observations of known shape $\alpha$ and unknown scale $\Theta$, an inverse-gamma distribution prior with shape $k$ and scale $\lambda$ gives an inverse-gamma distribution posterior with shape $k+n\alpha$ and scale $\lambda +{\sum }_j{x}_j$. The posterior mean of the conditional claim expected value $\alpha \Theta$ is $\alpha (\lambda +{\sum }_j{x}_j)/(k+n\alpha -1)$ when the denominator is positive.

With the gamma scale inverse-gamma conjugacy model and prior shape $k>2$, the Bühlmann model has $v/a=(k-1)/\alpha$. Its Bühlmann credibility premium equals $\alpha (\lambda +{\sum }_j{X}_j)/(k+n\alpha -1)$, exactly the Bayes estimator under squared error loss of $\alpha \Theta$. Equality holds because the posterior mean is already affine in the observed sample mean.

For a normal linear model with coefficient vector $\eta$ and residual variance ${\sigma }^2$, a normal-inverse-gamma prior has $\eta \mid {\sigma }^2\sim N(m,{\sigma }^{2}V)$ and ${\sigma }^2\sim \mathrm{IG}(a,b)$, with $V$ a positive-definite matrix and $a,b>0$. The inverse-gamma density is proportional to $({\sigma }^2{)}^{-a-1}{e}^{-b/{\sigma }^2}$. Multiplying by the normal linear model likelihood function and completing the square preserves the family, giving an analytic posterior and evidence integral.

For an exponential family with density proportional to $\exp \{{\theta }^{T}T(x)-A(\theta )\}$, its natural conjugate prior has density proportional to $\exp \{{\theta }^T{\lambda }_1-{\lambda }_{2}A(\theta )\}$. Observations update ${\lambda }_1$ by adding sufficient statistics and update ${\lambda }_2$ by adding sample size.

For a positive-support exponential family $f(x\mid \theta )=p(x)e^{-\theta x}/q(\theta )$, normalization gives $m(\theta )=-q^{\prime }(\theta )/q(\theta )$. The natural conjugate prior proportional to $q(\theta {)}^{-k}{e}^{-k\mu \theta }$ has logarithmic derivative $k(m(\theta )-\mu )$. If its endpoint density values vanish, integration gives prior expected value $\mathbb{E}[m(\Theta )]=\mu$. Observations update $k$ to $k+n$ and $\mu$ to $(k\mu +{\sum }_i{x}_i)/(k+n)$. The endpoint control for a Laplace-family conjugate posterior justifies the same mean identity after updating, giving an exact credibility estimate with credibility factor $n/(n+k)$.

In the positive-support exponential family $q(\theta )=\int p(x)e^{-\theta x}dx$, a proper natural conjugate prior with zero endpoint density values has mean parameter strictly inside the essential convex support $(a,b)$. Updating that mean by a positive-weight average with supported observations keeps it inside $(a,b)$. At finite parameter endpoints, prior vanishing forces $q\to \infty$, so the increased posterior power $q^{-k-n}$ still vanishes. At positive infinity, positive base-measure mass below a point $d$ smaller than the updated mean bounds $q(\theta )$ below by $C{e}^{-d\theta }$, giving exponential decay of the posterior kernel. Mass above the updated mean gives the analogous bound at negative infinity. Thus the updated density is proper and has the boundary condition required to integrate its score.