Exponential family

A density $h(y)\exp \{\theta T(y)-A(\theta )\}$ has sufficient statistic $T$ and cumulant function $A$.

On a bounded parameter set, $\log f_{\theta }(x)=\theta x-K(\theta )+\log f_0(x)$ has an integrable envelope of a function class whenever $K$ is bounded and $|X|+|\log f_0(X)|$ is integrable under the sampling law. A compact subset of the interior natural-parameter domain bounds the cumulant function of an exponential family and provides moments under model sampling; log-base-density integrability is an additional condition.

A conjugate prior belongs to a family whose posterior remains in that family after multiplication by the likelihood.

Conditionally independent gamma distributions with known shape $r$ and unknown rate $\Theta$ have likelihood proportional to ${\Theta }^{nr}{e}^{-\Theta {\sum }_i{x}_i}$. A gamma distribution prior of shape $A$ and rate $B$ therefore updates to shape $A+nr$ and rate $B+{\sum }_i{x}_i$. When $A+nr>1$, the posterior mean of the conditional claim mean $r/\Theta$ is $r(B+{\sum }_i{x}_i)/(A+nr-1)$. For $A=rk+1$ and $B=k\mu$, this is a credibility estimate with weight $n/(n+k)$. It is the reciprocal-rate version of gamma scale inverse-gamma conjugacy.

A Pareto distribution prior for the upper limit of independent uniform distributions remains Pareto after observing data. The likelihood adds the sample size to the shape and replaces the lower bound by the larger of the prior bound and sample maximum.

For positive observations and $m=\max (\beta ,{\max }_j{y}_j)$, the Bayesian model evidence follows by integrating the uniform distribution likelihood against a Pareto distribution prior. For independent groups with shared fixed hyperparameters, these evidences multiply. The resulting hyperparameter log likelihood is $I\log \alpha -{\sum }_i\log (\alpha +n_i)-\alpha {\sum }_i\log (m_i/\beta )$ up to a constant.

For conditionally independent gamma distribution observations of known shape $\alpha$ and unknown scale $\Theta$, an inverse-gamma distribution prior with shape $k$ and scale $\lambda$ gives an inverse-gamma distribution posterior with shape $k+n\alpha$ and scale $\lambda +{\sum }_j{x}_j$. The posterior mean of the conditional claim expected value $\alpha \Theta$ is $\alpha (\lambda +{\sum }_j{x}_j)/(k+n\alpha -1)$ when the denominator is positive.

With the gamma scale inverse-gamma conjugacy model and prior shape $k>2$, the Bühlmann model has $v/a=(k-1)/\alpha$. Its Bühlmann credibility premium equals $\alpha (\lambda +{\sum }_j{X}_j)/(k+n\alpha -1)$, exactly the Bayes estimator under squared error loss of $\alpha \Theta$. Equality holds because the posterior mean is already affine in the observed sample mean.

For a normal linear model with coefficient vector $\eta$ and residual variance ${\sigma }^2$, a normal-inverse-gamma prior has $\eta \mid {\sigma }^2\sim N(m,{\sigma }^{2}V)$ and ${\sigma }^2\sim \mathrm{IG}(a,b)$, with $V$ a positive-definite matrix and $a,b>0$. The inverse-gamma density is proportional to $({\sigma }^2{)}^{-a-1}{e}^{-b/{\sigma }^2}$. Multiplying by the normal linear model likelihood function and completing the square preserves the family, giving an analytic posterior and evidence integral.

For an exponential family with density proportional to $\exp \{{\theta }^{T}T(x)-A(\theta )\}$, its natural conjugate prior has density proportional to $\exp \{{\theta }^T{\lambda }_1-{\lambda }_{2}A(\theta )\}$. Observations update ${\lambda }_1$ by adding sufficient statistics and update ${\lambda }_2$ by adding sample size.

For a positive-support exponential family $f(x\mid \theta )=p(x)e^{-\theta x}/q(\theta )$, normalization gives $m(\theta )=-q^{\prime }(\theta )/q(\theta )$. The natural conjugate prior proportional to $q(\theta {)}^{-k}{e}^{-k\mu \theta }$ has logarithmic derivative $k(m(\theta )-\mu )$. If its endpoint density values vanish, integration gives prior expected value $\mathbb{E}[m(\Theta )]=\mu$. Observations update $k$ to $k+n$ and $\mu$ to $(k\mu +{\sum }_i{x}_i)/(k+n)$. The endpoint control for a Laplace-family conjugate posterior justifies the same mean identity after updating, giving an exact credibility estimate with credibility factor $n/(n+k)$.

In the positive-support exponential family $q(\theta )=\int p(x)e^{-\theta x}dx$, a proper natural conjugate prior with zero endpoint density values has mean parameter strictly inside the essential convex support $(a,b)$. Updating that mean by a positive-weight average with supported observations keeps it inside $(a,b)$. At finite parameter endpoints, prior vanishing forces $q\to \infty$, so the increased posterior power $q^{-k-n}$ still vanishes. At positive infinity, positive base-measure mass below a point $d$ smaller than the updated mean bounds $q(\theta )$ below by $C{e}^{-d\theta }$, giving exponential decay of the posterior kernel. Mass above the updated mean gives the analogous bound at negative infinity. Thus the updated density is proper and has the boundary condition required to integrate its score.

In the canonical representation$\theta$ is the natural parameter and $T$ is the natural statistic.

The natural parameter space is the set of $\theta$ for which the normalizing integral $\int h(y)e^{{\theta }^{T}T(y)}dy$ is finite. Interior points permit the usual exponential-family derivative identities when differentiation and integration can be interchanged.

The cumulant function, or log-partition function, is the normalizing termwith a sum in the discrete case.

Differentiating the normalization identity gives

The mean parameter corresponding to the natural parameter $\theta$ is

For a finite support whose convex hull has full dimension, every interior target mean is realized by a unique natural parameter. Minimize the coercive function $\log {\sum }_n{e}^{\theta \cdot n}-\theta \cdot \mu$; its gradient is the mean discrepancy and its Hessian matrix is the positive-definite covariance matrix of the support.

For an expected value parameter $\mu >0$ and a shape parameter $\lambda >0$, the inverse Gaussian distribution has probability density functionIts mean is $\mu$ and its variance is ${\mu }^3/\lambda$.

For $X\sim IG(\mu ,\lambda )$, $h(X)=\lambda (X-\mu {)}^2/({\mu }^{2}X)$ has the chi-squared-one law. Its two inverse branches contribute weights $\mu /(\mu +x_{-})$ and $\mu /(\mu +x_{+})$ to the transformed density, and reciprocal roots make those weights sum to one.

For a chi-squared-one variable $Y$, the positive inverse branches of $h(x)=\lambda (x-\mu {)}^2/({\mu }^{2}x)$ have product ${\mu }^2$. Weight a branch value $x$ by $\mu /(\mu +x)$; the two weights sum to one. Multiplying the chi-squared density by this weight and the branch Jacobian gives the inverse Gaussian density on both sides of $\mu$.

With $t=\mu Y/(2\lambda )$, the lower root is $\mu /(1+t+\sqrt{t(t+2)})$ and the upper root is ${\mu }^2/x_{-}$. Rationalizing the lower-root formula avoids subtracting nearly equal large numbers when $Y$ is large.

The deviance from ${\theta }_1$ to ${\theta }_2$ is twice the Kullback-Leibler divergence:

For fixed positive integer $r$, the failures-before-the-$r$th-success distribution is a one-parameter exponential family withand carrier $h(x)=\left(\genfrac{}{}{0ex}{}{x+r-1}{x}\right)$.

With dispersion $\varphi$ and known weight $w$, an exponential dispersion density has exponentIts mean is $b^{\prime }(\theta )$ and its variance is $(\varphi /w)b^{\prime \prime }(\theta )$.

The dispersion parameter scales the variance of an exponential dispersion family while leaving its mean function unchanged.

Underdispersion means less variation than prescribed by a reference statistical model, corresponding to a working dispersion parameter below one. It is not, by itself, evidence that the reference mean function is improved.

Overdispersion occurs when observed conditional variation exceeds that prescribed by a statistical model. For count data it commonly means variance greater than the mean assumed by a Poisson model.

The variance function writes an exponential dispersion family's variance as $\mathrm{Var}(Y)=\varphi V(\mu )/w$.