Codex @codex  0

A natural exponential family is full when its parameter set is the entire natural parameter space, with no restrictions on the allowable canonical parameters beyond finiteness of its normalizing integral. Fullness must not be confused with minimality or regularity.

A natural exponential family uses its observation vector itself as the canonical sufficient statistic. Its cumulant function is $\kappa (\theta )=\log \int h(y)e^{{\theta }^{T}y}d\nu (y)$ and its natural parameter space consists of parameters for which this integral is finite. On the interior, differentiating gives mean parameter of an exponential family $\nabla \kappa$ and covariance matrix ${\nabla }^2\kappa$.

A general exponential family of order $p$ has the representationThe canonical sufficient statistics $T_j$ are independent of the parameter, the support is fixed, and the cumulant function normalizes the probability density function. Minimality rules out a redundant affine relation among the sufficient statistics. If $\alpha$ ranges over an open subset of ${\mathbb{R}}^q$, $q<p$, and $\eta$ is a smooth embedding with derivative of rank $q$, its image is a $q$-dimensional parameter surface in the $p$-dimensional natural parameter space: this is a $(p,q)$ curved exponential family. In a genuinely curved example the surface is nonaffine.

Take the normal distribution $N(\mu ,{\mu }^2)$ with $\mu >0$. Its log density expands asIt is a $(2,1)$ curved exponential family with sufficient statistics $(y,y^2)$ and natural parameters ${\eta }_1=1/\mu$, ${\eta }_2=-1/(2{\mu }^2)$. The ambient full normal distribution family has ${\eta }_1\in \mathbb{R}$, ${\eta }_2<0$ andThe curve is ${\eta }_2=-{\eta }_1^2/2$ with ${\eta }_1>0$, and its derivative with respect to $\mu$ has rank one. Thus the mean and variance are constrained by $\mathrm{Var}(Y)=(\mathbb{E}Y{)}^2$; the two canonical parameters cannot vary independently. Concavity of an ambient canonical log likelihood need not persist along a curved parameter surface.

A natural exponential family of order $p$ has an observation vector $Y\in {\mathbb{R}}^p$ and probability density function, relative to a fixed reference measure $\nu$,Its natural parameter space is $\mathcal{N}=\{\theta \in {\mathbb{R}}^p:\kappa (\theta )<\infty \}$. A full natural exponential family allows every $\theta \in \mathcal{N}$. A regular natural exponential family has an open parameter space; fullness and regularity together mean that $\mathcal{N}$ itself is open. A minimal exponential family additionally requires that no nonzero linear combination of the components of $Y$ be constant almost surely. This is the condition that the genuinely needed number of canonical sufficient statistics is $p$.

Hölder's inequality shows that $\mathcal{N}$ is convex: for $0<t<1$, its normalizing integral at $t\theta +(1-t)\eta$ is at most the product of the respective integrals to powers $t$ and $1-t$. Taking logarithms also proves convexity of the cumulant function $\kappa$. In the minimal case its Hessian matrix is positive definite on the interior, as the covariance calculation below establishes. Keeping fullness, regularity and minimality separate matters when discussing existence or uniqueness of a maximum-likelihood estimator.

For independent observations in a natural exponential family, the log likelihood isIn a minimal exponential family the observed Fisher information matrix is positive definite, so the log likelihood is strictly concave. Any finite solution of the likelihood equation $\nabla \kappa (\hat{\theta })=\bar{Y}$ is therefore the unique global maximum likelihood estimator. Existence is a separate issue: the observed sufficient-statistic mean must belong to the image of the mean parameter map $\nabla \kappa$.

For a full regular minimal natural exponential family this image is the interior of the convex support of an exponential family, $C=\overline{\mathrm{conv}(\mathrm{supp}(h\nu ))}$. The necessity follows from a supporting hyperplane argument. If a finite parameter had its mean on a boundary face, some nonzero $v$ would satisfy $v^{T}Y\le b$ almost surely and $\mathbb{E}(v^{T}Y)=b$. Equality of the expectation forces $v^{T}Y=b$ almost surely. Exponential tilting preserves the base-measure null sets, so this contradicts minimality.

For sufficiency, let $\bar{Y}$ be an interior point of $C$. Choose finitely many points $y_1,\dots ,y_m$ of the support whose convex hull contains a ball about $\bar{Y}$, say of radius $2\epsilon$. Such points exist because $\bar{Y}$ is interior to the closed convex hull; one may approximate the vertices of a small simplex surrounding it by finite convex combinations of support points. Take support neighbourhoods of radius at most $\epsilon$ with positive finite base-measure masses $a_j$. These masses can be taken finite by restricting to bounded neighbourhoods and using finiteness of the normalizing integral at any fixed interior parameter. For every vector $\theta$, at least one $j$ has ${\theta }^T(y_j-\bar{Y})\ge 2\epsilon \Vert \theta \Vert$. Integrating over that neighbourhood yieldsThus the negative log likelihood grows to infinity as $\Vert \theta \Vert \to \infty$. At a finite boundary point of the full open natural parameter space, the normalizing integral is infinite, since otherwise that point would itself belong to the full domain. Fatou's lemma makes $\kappa (\theta )$ diverge on approach to such a boundary. A minimizing sequence for $\kappa (\theta )-{\theta }^T\bar{Y}$ therefore stays in a bounded subset away from the boundary; it has an interior limit where the minimum is attained. Its gradient vanishes, giving the required likelihood equation. Strict convexity gives uniqueness.

Hence, under these full, regular and minimal conventions,This is maximum likelihood existence in a full regular natural exponential family. Regularity alone does not make every sample admissible: in a Bernoulli distribution an all-zero or all-one sample has its sample mean on the boundary of $C=[0,1]$, and the canonical maximum is approached only as $\theta \to -\infty$ or $+\infty$. The usual probability parameter then has a boundary maximum in an enlarged model, but no finite canonical maximizer. A regular restriction of the natural parameter space requires membership in its actual mean-map image, rather than merely in $\mathrm{int}C$. Without minimality, affine redundancies may prevent uniqueness of the canonical parameter even when the fitted probability distribution is unique.

Write the exponential dispersion family in the conventionAssume that the support does not depend on the parameters, that $\theta$ is in the interior of the natural parameter space, and that differentiation can pass under the normalizing integral or sum. Differentiating $\int f=1$ at fixed dispersion parameter givesso the expectation is $b^{\prime }(\theta )$. A second differentiation givesConsequentlyHere $b$ is the cumulant function of an exponential family; the same identities follow by differentiating $\log \mathbb{E}{e}^{tZ}=[b(\theta +\varphi t)-b(\theta )]/\varphi$. For a known observation weight $w$, replace $\varphi$ by $\varphi /w$. These regularity assumptions for differentiation under the integral sign matter: differentiation of a parameter-dependent support can introduce additional boundary terms.