Exponential family

A density $h(y)\exp \{\theta T(y)-A(\theta )\}$ has sufficient statistic $T$ and cumulant function $A$.

A $(p,q)$ curved exponential family, with $q<p$, restricts a $p$-parameter canonical family to a smooth rank-$q$ parameter surface, usually a nonaffine surface. For example, $N(\mu ,{\mu }^2)$ with $\mu >0$ has sufficient statistics $(Y,Y^2)$ and natural parameters $(1/\mu ,-1/(2{\mu }^2))$, constrained by ${\eta }_2=-{\eta }_1^2/2$. Canonical concavity in the ambient parameters need not imply concavity along the curved parameter surface.

This one-parameter normal distribution model has positive mean equal to its standard deviation. It is a curved exponential family: both the linear and quadratic natural parameters depend on the same scalar statistical parameter. It combines information about the center and spread, so the sample mean alone does not use all available Fisher information.

For a statistical sample of size $n$, let $A={\sum }_i{x}_i$ and $B={\sum }_i{x}_i^2$. Up to an additive constant the log-likelihood function is $\ell (\mu )=-n\log \mu -B/(2{\mu }^2)+A/\mu$. Its score function is $(B-A\mu -n{\mu }^2)/{\mu }^3$. When $B>0$, the numerator changes from positive to negative at its unique positive root, so the global maximum-likelihood estimator isThe all-zero observation vector has zero probability and is an exceptional case with an unbounded likelihood function near zero. Since $\mathbb{E}A=n\mu$ and $\mathbb{E}B=2n{\mu }^2$, the expected Fisher information is $I_n(\mu )=3n/{\mu }^2$. Thus regular asymptotic normality of a maximum likelihood estimator gives $\sqrt{n}(\hat{\mu }-\mu )\stackrel{d}{\to }N(0,{\mu }^2/3)$. The optimal asymptotic combination of a normal sample mean and standard deviation attains the same first-order precision.

For independent $N(\mu ,{\mu }^2)$ observations with $\mu >0$, the sufficient statistics are their sum and squared norm. Condition on $a=\sqrt{n}\Vert y\Vert /{\sum }_i{y}_i>0$ and write $v=\hat{\mu }=\Vert y\Vert /(q\sqrt{n})$, where $q=(\sqrt{1+4a^2}+1)/(2a)$. Polar coordinates supply radial measure $\Vert y{\Vert }^{n-1}d\Vert y\Vert$, while the conditional angular factor does not depend on $\mu$. This proves the displayed shape. Its normalization depends on $a,n$ but not $\mu$ after the substitution $v=\mu z$, and agrees with the normalized P-star approximation.

An order-$p$ exponential family is minimal if no nonzero linear combination of its $p$ canonical sufficient statistics is almost surely constant under the base measure. Equivalently, its sufficient-statistic support lies in no proper affine hyperplane. Its cumulant function then has positive-definite Hessian matrix on the interior natural domain, because that Hessian is a covariance matrix.

A natural exponential family uses its observation vector itself as the canonical sufficient statistic. Its cumulant function is $\kappa (\theta )=\log \int h(y)e^{{\theta }^{T}y}d\nu (y)$ and its natural parameter space consists of parameters for which this integral is finite. On the interior, differentiating gives mean parameter of an exponential family $\nabla \kappa$ and covariance matrix ${\nabla }^2\kappa$.

The convex support is the closed convex hull of the canonical sufficient-statistic support. For a minimal exponential family it has nonempty interior. Means at finite interior natural parameters lie in that interior, because a supporting hyperplane at a boundary mean would force the sufficient statistic itself onto that hyperplane.

A natural exponential family is regular when its parameter space is open. In a full regular family, differentiation under the normalizing integral is justified locally by exponential moments in a neighbourhood. Regularity does not guarantee that every possible data set has a finite maximum-likelihood estimator.

A natural exponential family is full when its parameter set is the entire natural parameter space, with no restrictions on the allowable canonical parameters beyond finiteness of its normalizing integral. Fullness must not be confused with minimality or regularity.

In a full regular minimal exponential family, a finite maximum-likelihood estimator exists precisely when the observed sufficient-statistic mean is in the interior of the convex support of an exponential family. It is unique because the log likelihood is strictly concave. For an interior mean, positive base-measure mass near finitely many support points surrounding the mean makes $\kappa (\theta )-{\theta }^T\bar{Y}$ grow at least linearly as $\Vert \theta \Vert \to \infty$. At a finite boundary of the full open natural domain, the normalizing integral diverges. Thus the likelihood attains its maximum in the domain. On a boundary face, no finite parameter can have the required mean. A regular restriction should instead use the exact mean-image criterion. These full-family conventions are also discussed in Geyer's exponential-family lecture notes.

For independent $N(\mu ,v)$ observations with unknown mean and variance, the first natural parameter is ${\varphi }_1=\mu /v$. Its maximum-likelihood estimator is $\bar{X}/[n^{-1}{\sum }_i(X_i-\bar{X}{)}^2]$. The delta method and independent asymptotic mean/variance errors give limiting variance $v^{-1}+2{\mu }^2{v}^{-2}$ for $\sqrt{n}({\hat{\varphi }}_1-{\varphi }_1)$. Knowing $v$ removes the second contribution and makes the estimator exactly normal with variance $1/(nv)$.

On a bounded parameter set, $\log f_{\theta }(x)=\theta x-K(\theta )+\log f_0(x)$ has an integrable envelope of a function class whenever $K$ is bounded and $|X|+|\log f_0(X)|$ is integrable under the sampling law. A compact subset of the interior natural-parameter domain bounds the cumulant function of an exponential family and provides moments under model sampling; log-base-density integrability is an additional condition.

A conjugate prior belongs to a family whose posterior remains in that family after multiplication by the likelihood.

A normal-gamma conjugate prior for the mean $\mu$ and precision parameter $\tau$ of a Gaussian distribution specifies a Gamma distribution $\tau \sim \mathrm{Gamma}(\alpha ,\beta )$ in the shape-rate convention and a conditional Gaussian distribution $\mu \mid \tau \sim N({\mu }_0,(K\tau {)}^{-1})$. Its joint probability density function is proportional to ${\tau }^{\alpha -1}\sqrt{\tau }\exp [-\beta \tau -K\tau (\mu -{\mu }_0{)}^2/2]$, with $\alpha ,\beta ,K>0$. For $n$ independent observations with Gaussian distribution $N(\mu ,{\tau }^{-1})$, multiplication by the likelihood and completing the square givesand ${\beta }_n=\beta +{\sum }_i(x_i-\bar{x}{)}^2/2+Kn(\bar{x}-{\mu }_0{)}^2/[2(K+n)]$. The identity behind conjugacy is $K(\mu -{\mu }_0{)}^2+{\sum }_i(x_i-\mu {)}^2=(K+n)(\mu -{\mu }_n{)}^2+{\sum }_i(x_i-\bar{x}{)}^2+Kn(\bar{x}-{\mu }_0{)}^2/(K+n)$.

Conditionally independent gamma distributions with known shape $r$ and unknown rate $\Theta$ have likelihood proportional to ${\Theta }^{nr}{e}^{-\Theta {\sum }_i{x}_i}$. A gamma distribution prior of shape $A$ and rate $B$ therefore updates to shape $A+nr$ and rate $B+{\sum }_i{x}_i$. When $A+nr>1$, the posterior mean of the conditional claim mean $r/\Theta$ is $r(B+{\sum }_i{x}_i)/(A+nr-1)$. For $A=rk+1$ and $B=k\mu$, this is a credibility estimate with weight $n/(n+k)$. It is the reciprocal-rate version of gamma scale inverse-gamma conjugacy.

A Pareto distribution prior for the upper limit of independent uniform distributions remains Pareto after observing data. The likelihood adds the sample size to the shape and replaces the lower bound by the larger of the prior bound and sample maximum.

For positive observations and $m=\max (\beta ,{\max }_j{y}_j)$, the Bayesian model evidence follows by integrating the uniform distribution likelihood against a Pareto distribution prior. For independent groups with shared fixed hyperparameters, these evidences multiply. The resulting hyperparameter log likelihood is $I\log \alpha -{\sum }_i\log (\alpha +n_i)-\alpha {\sum }_i\log (m_i/\beta )$ up to a constant.

For conditionally independent gamma distribution observations of known shape $\alpha$ and unknown scale $\Theta$, an inverse-gamma distribution prior with shape $k$ and scale $\lambda$ gives an inverse-gamma distribution posterior with shape $k+n\alpha$ and scale $\lambda +{\sum }_j{x}_j$. The posterior mean of the conditional claim expected value $\alpha \Theta$ is $\alpha (\lambda +{\sum }_j{x}_j)/(k+n\alpha -1)$ when the denominator is positive.

With the gamma scale inverse-gamma conjugacy model and prior shape $k>2$, the Bühlmann model has $v/a=(k-1)/\alpha$. Its Bühlmann credibility premium equals $\alpha (\lambda +{\sum }_j{X}_j)/(k+n\alpha -1)$, exactly the Bayes estimator under squared error loss of $\alpha \Theta$. Equality holds because the posterior mean is already affine in the observed sample mean.

For a normal linear model with coefficient vector $\eta$ and residual variance ${\sigma }^2$, a normal-inverse-gamma prior has $\eta \mid {\sigma }^2\sim N(m,{\sigma }^{2}V)$ and ${\sigma }^2\sim \mathrm{IG}(a,b)$, with $V$ a positive-definite matrix and $a,b>0$. The inverse-gamma density is proportional to $({\sigma }^2{)}^{-a-1}{e}^{-b/{\sigma }^2}$. Multiplying by the normal linear model likelihood function and completing the square preserves the family, giving an analytic posterior and evidence integral.

For an exponential family with density proportional to $\exp \{{\theta }^{T}T(x)-A(\theta )\}$, its natural conjugate prior has density proportional to $\exp \{{\theta }^T{\lambda }_1-{\lambda }_{2}A(\theta )\}$. Observations update ${\lambda }_1$ by adding sufficient statistics and update ${\lambda }_2$ by adding sample size.

For a positive-support exponential family $f(x\mid \theta )=p(x)e^{-\theta x}/q(\theta )$, normalization gives $m(\theta )=-q^{\prime }(\theta )/q(\theta )$. The natural conjugate prior proportional to $q(\theta {)}^{-k}{e}^{-k\mu \theta }$ has logarithmic derivative $k(m(\theta )-\mu )$. If its endpoint density values vanish, integration gives prior expected value $\mathbb{E}[m(\Theta )]=\mu$. Observations update $k$ to $k+n$ and $\mu$ to $(k\mu +{\sum }_i{x}_i)/(k+n)$. The endpoint control for a Laplace-family conjugate posterior justifies the same mean identity after updating, giving an exact credibility estimate with credibility factor $n/(n+k)$.

In the positive-support exponential family $q(\theta )=\int p(x)e^{-\theta x}dx$, a proper natural conjugate prior with zero endpoint density values has mean parameter strictly inside the essential convex support $(a,b)$. Updating that mean by a positive-weight average with supported observations keeps it inside $(a,b)$. At finite parameter endpoints, prior vanishing forces $q\to \infty$, so the increased posterior power $q^{-k-n}$ still vanishes. At positive infinity, positive base-measure mass below a point $d$ smaller than the updated mean bounds $q(\theta )$ below by $C{e}^{-d\theta }$, giving exponential decay of the posterior kernel. Mass above the updated mean gives the analogous bound at negative infinity. Thus the updated density is proper and has the boundary condition required to integrate its score.

In the canonical representation$\theta$ is the natural parameter and $T$ is the natural statistic.

The natural parameter space is the set of $\theta$ for which the normalizing integral $\int h(y)e^{{\theta }^{T}T(y)}dy$ is finite. Interior points permit the usual exponential-family derivative identities when differentiation and integration can be interchanged.

The cumulant function, or log-partition function, is the normalizing termwith a sum in the discrete case.

Differentiating the normalization identity gives

The mean parameter corresponding to the natural parameter $\theta$ is

For a finite support whose convex hull has full dimension, every interior target mean is realized by a unique natural parameter. Minimize the coercive function $\log {\sum }_n{e}^{\theta \cdot n}-\theta \cdot \mu$; its gradient is the mean discrepancy and its Hessian matrix is the positive-definite covariance matrix of the support.

For an expected value parameter $\mu >0$ and a shape parameter $\lambda >0$, the inverse Gaussian distribution has probability density functionIts mean is $\mu$ and its variance is ${\mu }^3/\lambda$.

For an independent statistical sample from an Inverse Gaussian distribution with mean $\mu >0$ and known shape $\lambda >0$, the maximum-likelihood estimator of $\mu$ is the sample mean. To prove this, expand the log-likelihood function to get $\ell (\mu )=C-\lambda {\sum }_i{y}_i/(2{\mu }^2)+n\lambda /\mu$. Its score function is $\lambda ({\sum }_i{y}_i-n\mu )/{\mu }^3$, positive below $\bar{y}$ and negative above it. Taking the negative expected second derivative gives one-observation Fisher information $I_1(\mu )=\lambda /{\mu }^3$. Thus the regular asymptotic normality of a maximum likelihood estimator yields $\sqrt{n}(\bar{Y}-\mu )\stackrel{d}{\to }N(0,{\mu }^3/\lambda )$. The same conclusion follows from the central limit theorem and the distribution's variance ${\mu }^3/\lambda$.

For the Inverse Gaussian distribution parametrized by shape $\psi$ and mean $\lambda$, the joint maximum-likelihood estimators are $\hat{\lambda }=\bar{Y}$ and $\hat{\psi }=(\overline{Y^{-1}}-1/\bar{Y}{)}^{-1}$. With known $\lambda$, the shape estimator is instead $(\overline{Y^{-1}}-2/\lambda +\bar{Y}/{\lambda }^2{)}^{-1}$. Parameter orthogonality gives the same Wald test formula $n(\hat{\psi }-{\psi }_0{)}^2/(2{\hat{\psi }}^2)$ in both fits. The two estimators differ by $O_p(n^{-1})$ under regular model sampling, so the test statistics agree to first order under the null; they need not be numerically equal in finite samples.

In the parametrization with density proportional to $y^{-3/2}\exp [-(\lambda /y+\varphi y)/2]$, the Inverse Gaussian distribution has cumulant-generating function $K(t)=\sqrt{\lambda \varphi }-\sqrt{\lambda (\varphi -2t)}$. For $n$ independent copies, the average has this same family with parameters $(n\varphi ,n\lambda )$, while the sum has parameters $(\varphi ,n^2\lambda )$. These assertions follow by multiplying moment-generating functions and scaling the density. The more usual mean parameter is $\mu =\sqrt{\lambda /\varphi }$, so the sum has mean $n\mu$ and shape $n^2\lambda$.

For the inverse Gaussian sum closure, take $x=s/n>0$. The saddle solves $\hat{t}=(\varphi -\lambda /x^2)/2$, and $K^{\prime \prime }(\hat{t})=x^3/\lambda$. Substitution in the saddlepoint density approximation givesThis is the exact sum density, so its leading saddlepoint expression needs no correction or renormalization.

For $X\sim IG(\mu ,\lambda )$, $h(X)=\lambda (X-\mu {)}^2/({\mu }^{2}X)$ has the chi-squared-one law. Its two inverse branches contribute weights $\mu /(\mu +x_{-})$ and $\mu /(\mu +x_{+})$ to the transformed density, and reciprocal roots make those weights sum to one.

For a chi-squared-one variable $Y$, the positive inverse branches of $h(x)=\lambda (x-\mu {)}^2/({\mu }^{2}x)$ have product ${\mu }^2$. Weight a branch value $x$ by $\mu /(\mu +x)$; the two weights sum to one. Multiplying the chi-squared density by this weight and the branch Jacobian gives the inverse Gaussian density on both sides of $\mu$.

With $t=\mu Y/(2\lambda )$, the lower root is $\mu /(1+t+\sqrt{t(t+2)})$ and the upper root is ${\mu }^2/x_{-}$. Rationalizing the lower-root formula avoids subtracting nearly equal large numbers when $Y$ is large.

The deviance from ${\theta }_1$ to ${\theta }_2$ is twice the Kullback-Leibler divergence:

For an exponential dispersion family with a common dispersion parameter $\varphi$, scaled deviance compares a fitted model with the saturated statistical model at the same dispersion:For a normal distribution with variance ${\sigma }^2$, unscaled deviance is the residual sum of squares and scaled deviance is that sum divided by ${\sigma }^2$. For the binomial distribution, $\varphi =1$ and the two coincide. Unknown Gaussian dispersion requires variance estimation; raw deviance differences should not be treated as chi-squared statistics without the scale.

For fixed positive integer $r$, the failures-before-the-$r$th-success distribution is a one-parameter exponential family withand carrier $h(x)=\left(\genfrac{}{}{0ex}{}{x+r-1}{x}\right)$.

With dispersion $\varphi$ and known weight $w$, an exponential dispersion density has exponentIts mean is $b^{\prime }(\theta )$ and its variance is $(\varphi /w)b^{\prime \prime }(\theta )$.

An order-one exponential dispersion family has a scalar natural parameter and canonical sufficient statistic $y$; its density or mass function has the displayed form on parameter-independent support. Differentiating its normalization gives $\mathbb{E}Y=b^{\prime }(\theta )$ and $\mathrm{Var}(Y)=\varphi b^{\prime \prime }(\theta )$. Known observation weights $w$ replace $\varphi$ by $\varphi /w$. Its variance function is $V(\mu )=b^{\prime \prime }((b^{\prime }{)}^{-1}(\mu ))$, and its canonical link function is $(b^{\prime }{)}^{-1}$.

The dispersion parameter scales the variance of an exponential dispersion family while leaving its mean function unchanged.

Underdispersion means less variation than prescribed by a reference statistical model, corresponding to a working dispersion parameter below one. It is not, by itself, evidence that the reference mean function is improved.

Overdispersion occurs when observed conditional variation exceeds that prescribed by a statistical model. For count data it commonly means variance greater than the mean assumed by a Poisson model.

The variance function writes an exponential dispersion family's variance as $\mathrm{Var}(Y)=\varphi V(\mu )/w$.