Complete sufficient statistic Created 2026-10-07 Updated 2026-10-14
A sufficient statistic is complete when every integrable function of it with expectation zero for every parameter is almost surely zero for every parameter. Full regular one-parameter exponential families with natural parameter ranging over an open interval provide a standard completeness result. Completeness makes an unbiased function of a sufficient statistic unique, giving the Lehmann–Scheffé theorem.
Curved exponential family 2026-10-14
A curved exponential family, with , restricts a -parameter canonical family to a smooth rank- parameter surface, usually a nonaffine surface. For example, with has sufficient statistics and natural parameters , constrained by . Canonical concavity in the ambient parameters need not imply concavity along the curved parameter surface.
In a regular one-parameter statistical model, equality in the Cramér-Rao lower bound for an unbiased estimator implies the score function . Integrating in the parameter on a regular connected interval gives , with and . Thus efficiency for every parameter value forces an exponential family structure, under the usual common-support and differentiation assumptions.
In the positive-support exponential family , a proper natural conjugate prior with zero endpoint density values has mean parameter strictly inside the essential convex support . Updating that mean by a positive-weight average with supported observations keeps it inside . At finite parameter endpoints, prior vanishing forces , so the increased posterior power still vanishes. At positive infinity, positive base-measure mass below a point smaller than the updated mean bounds below by , giving exponential decay of the posterior kernel. Mass above the updated mean gives the analogous bound at negative infinity. Thus the updated density is proper and has the boundary condition required to integrate its score.
For known size , the negative binomial distribution is an exponential family with natural parameter and cumulant . A generalized linear model can use a logarithmic link function for its mean, even though that link is not canonical. Jointly estimating the size changes the family and variance function, requiring additional updates beyond a single fixed-family GLM fit; this is what MASS::glm.nb does.
For a finite list of real values , maximize information entropy with , , and . Lagrange multipliers give
The constrained second variation is for any nonzero tangent perturbation, proving uniqueness of the entropy maximum whenever a feasible interior solution exists. The mean has derivative , so for nonconstant it decreases strictly with and ranges between the largest and smallest as ranges over the real line. Thus a finite exists uniquely exactly when .
At an extreme mean, strictly positive probabilities are impossible unless all coincide. If all , the mean constraint is redundant when , the maximum is uniform, and is not identifiable. This finite exponential family is an exponential tilting of the uniform distribution.
An order- exponential family is minimal if no nonzero linear combination of its canonical sufficient statistics is almost surely constant under the base measure. Equivalently, its sufficient-statistic support lies in no proper affine hyperplane. Its cumulant function then has positive-definite Hessian matrix on the interior natural domain, because that Hessian is a covariance matrix.
For a positive-support exponential family , normalization gives . The natural conjugate prior proportional to has logarithmic derivative . If its endpoint density values vanish, integration gives prior expected value . Observations update to and to . The endpoint control for a Laplace-family conjugate posterior justifies the same mean identity after updating, giving an exact credibility estimate with credibility factor .
Natural conjugate prior Created 2026-09-28 Updated 2026-10-05
For an exponential family with density proportional to , its natural conjugate prior has density proportional to . Observations update by adding sufficient statistics and update by adding sample size.
A natural exponential family uses its observation vector itself as the canonical sufficient statistic. Its cumulant function is and its natural parameter space consists of parameters for which this integral is finite. On the interior, differentiating gives mean parameter of an exponential family and covariance matrix .
A finite-dimensional exponential family has common dominating measure and density
Here is the natural parameter of an exponential family, and normalization determines the cumulant function of an exponential family . A full regular family lets vary over an open natural parameter domain. A curved exponential family restricts to a lower-dimensional submanifold, while a minimal representation removes redundant affine combinations of the statistic.
Differentiating the normalization under regularity gives
Consequently is convex and its Hessian is the Fisher information matrix in natural coordinates. For an independent size- sample, is a sufficient statistic: the joint density factors into times , so the conditional law given loses its parameter dependence. An interior maximum-likelihood estimator solves
matching empirical and theoretical sufficient-statistic means. Under minimality the Hessian is positive definite, so the log-likelihood is strictly concave; existence of an interior maximizer still needs the observed statistic to be in the appropriate interior convex support.
For a full open natural family, the canonical sufficient statistic is a complete statistic: if throughout the open domain, exponential tilting about an interior point makes the Laplace transform of the integrable signed measure zero on a neighborhood. Transform uniqueness makes that measure zero, so almost surely. This requires integrability under the family. It is not an automatic property of curved restrictions. The sufficient-statistic reduction, explicit information and cumulant identities make these families especially useful for exact and higher-order inference. Normal distributions, Poisson distributions and gamma distributions supply familiar examples.
A transformation model is built differently: a group acts on the sample and parameter spaces, and its probability laws satisfy
In a transitive model all laws are transforms of a fixed reference law. The location family is generated by translations, with density . The positive-scale family has density , and the location-scale family combines them as
The Jacobian factor is essential. For independent observations its power is . The base density need not have moments, and a transformation model need not have fixed support.
An equivariant estimator respects the parameter action. With a well-defined equivariant fit valued in the transformation group, the normalized sample
is a maximal invariant. Equivariance gives ; if normalized samples agree, is obtained from by . In a transitive transformation model its law is parameter-free, so it is an ancillary statistic. For location and positive scale, centered standardized residuals provide this construction on nondegenerate samples. If the parameter action has a nontrivial stabilizer, normalization must retain its quotient action rather than pretending that a unique group element is determined.
This geometry motivates conditioning on an ancillary and using the P-star approximation for the fitted parameter. Ancillarity means parameter-free distribution; it does not by itself assert independence from the estimator. In a normal location model with known scale, the sample mean and centered residuals are independent by their joint Gaussian law and zero covariance. In an exponential scale model, the sample sum and its normalized proportions are independent, again an additional model property.
The two classes overlap but are distinct. A normal location-scale family is both an exponential family and a transformation model; an exponential mean family is both a one-dimensional natural exponential family and a scale model. A general Cauchy distribution location family is a transformation model without a fixed finite-dimensional regular exponential-family sufficient-statistic reduction. Exponential-family structure emphasizes sufficient statistics and natural-parameter convexity; transformation structure emphasizes equivariance, group orbits and ancillary reduction.
Consider a regular -parameter statistical model for which the data can locally be expressed as , where is the maximum-likelihood estimator and is a suitable exact or higher-order ancillary statistic. Write the log-likelihood as , and let
be its fitted observed information. The P-star approximation to the conditional sampling density of , given , is
The normalizing factor is independent of and is chosen so the density integrates to one over the feasible estimator coordinates. Its leading Gaussian value is , but exact normalization can change it. Both the fitted likelihood and the fitted information vary with ; the observed dataset is not held fixed while integrating over this sampling coordinate.
To see the structure, expand the likelihood difference for close to . Its leading term is minus one half of the information-weighted quadratic displacement, while supplies the local density scale. This recovers the first-order multivariate normal distribution of an efficient maximum-likelihood estimator. The formula is invariant under smooth one-to-one parameter changes: at a likelihood maximum the score vanishes, so the Hessian transforms as a quadratic form. Its square-root determinant supplies exactly the Jacobian determinant required for the estimator density to transform.
There is a direct derivation in a full regular exponential family. Let be the sufficient statistic for an independent sample, with likelihood
The maximum-likelihood equation is , and its derivative is . Applying a multivariate saddlepoint density approximation to gives
Changing variables from to multiplies by , producing the displayed P-star form. In more general models, ancillary conditioning supplies the relevant local sample coordinates; an ancillary is not an arbitrary extra statistic.
For an illustrative exact case, take an independent sample from an exponential distribution with mean , so . The fitted observed information is , and
Thus P-star has density shape . Normalizing gives
which is exactly the gamma distribution sampling density of the mean. Here sample ratios provide ancillary coordinates and are independent of the mean.
In suitable regular models the unnormalized leading construction has relative error of order , and higher-order ancillary constructions with normalization can attain relative order . Such accuracy needs the stated smoothness and ancillary hypotheses; it is not a universal consequence of asymptotic normality. The exact scale-model example is likewise a special property, not a general identity. The formula is useful for refined conditional inference, score distributions, and nuisance-likelihood adjustments, including modified profile likelihood. It describes a sampling density of the estimator, rather than a posterior density of the parameter.
The profile likelihood for the parameter of interest replaces the nuisance parameter by its constrained best fit: , with . It is not in general a normalized sampling density. A conditional likelihood instead uses the sampling law conditional on a statistic carrying the nuisance dependence. In the canonical exponential family here, conditioning on removes because the factor involving is constant over the conditioning fibre.
Let and let the observed equal . For the cumulant-generating function of at fixed ,
The saddlepoint solves , equivalently with . This is also the constrained likelihood equation, since . Assume an interior saddle, , and the usual smooth nonlattice conditions under which has a density. Exponential tilting to makes the tilted mean. A local normal approximation gives tilted density at that mean; undoing the tilt gives the saddlepoint density approximation
The factor is for the sample mean, rather than the sum. A transform-based derivation gives the same expression by expanding to second order at its stationary point.
The joint log-likelihood is . Subtracting cancels every appearance of the original , leaving
Any change from the ambient density to a conditional density on contributes only a data-dependent Jacobian determinant. Thus, up to a constant independent of , the saddlepoint conditional likelihood adjustment is
Equivalently up to the constant , where is the nuisance observed information. The adjustment has a positive sign. The printed exponential-family form alone does not guarantee a density for : for example a continuous can have . In a lattice case one must approximate the probability mass of the sum instead; the displayed density calculation uses the regular continuous interpretation.
Put , and . Apart from a data-only constant, the log-likelihood is
The natural parameters in the ambient two-parameter exponential family of normal distributions are and , restricted to the nonaffine curve , . Its parametrization has rank one. Thus this is a one-dimensional curved exponential family inside a two-dimensional full family.
The Fisher-Neyman factorization theorem makes sufficient. For two samples , the parameter-dependent part of their log density ratio is . A polynomial in is constant on only if both nonconstant coefficients vanish. The likelihood-ratio criterion for minimal sufficiency therefore proves is minimal sufficient.
Write with independent . Since , has a probability distribution independent of : it is an ancillary statistic. The denominator is nonzero almost surely. The Cauchy-Schwarz inequality gives ; on the requested positive branch, and .
The likelihood equation is . Its positive root is
There is exactly one positive root. The sign of the score function changes from positive to negative there, and at both ends of when , so it is the unique maximum-likelihood estimator. The useful identities are
Substitution yields, dropping the constant ,
Differentiating twice and using at the fitted value gives the observed information
The scalar P-star approximation uses . Consequently, with its normalizing factor explicitly retained,
Here is chosen to make the density integrate to one, and is independent of . To see this without assuming an approximate normalizer, define
The integral is finite and positive, and normalized is exactly
Indeed .
There is a direct check of this conditional scale density in a curved Gaussian family. In polar coordinates , the sample density times volume element is proportional to . Conditioning on fixes . The remaining angular factor is independent of , while . Thus the exact conditional radial density has the same normalized form. For the positive ancillary branch fixes the positive sign and the same one-dimensional radial calculation applies. In this scale model the normalized expression equals the exact conditional density, not merely its leading approximation.
For , compute its conditional distribution by one-dimensional quadrature:
It is zero for ; with the conditional-density calculation just given, this normalized formula is exact for almost every admissible . Numerically, subtract the maximum log integrand before exponentiating to avoid overflow. The sampling probability still depends on the stipulated true ; replacing it by a fitted value would be a separate plug-in inference step.
Differentiate the normalization of the exponential family density with respect to , assuming the regularity that permits differentiation under its integral. The density derivative is . Consequently
Differentiating the expectation once more gives , so
The log-likelihood derivative with respect to is . For the generalized linear model with linear predictor , the chain rule gives
Using independence to add the log-likelihood contributions proves the score function
Here includes the dispersion parameter . Defining instead a unit-dispersion variance function would require retaining an explicit in the denominator.
The Poisson distribution mass function is
It is an exponential dispersion family of order one with
Indeed, and . A generalized linear model specifies independent random variables from a response exponential family, together with a linear predictor and a link function relating it to the response mean. The Poisson canonical link is .
For the insurance counts, let class and merit be the reference levels in a regression factor. The fitted Poisson regression assumes
The insured exposure is known; its logarithm is an offset, not a coefficient to estimate. The model has eight free mean parameters and fixes the dispersion parameter at one. It assumes additive class and merit effects on the log-rate scale, hence multiplicative effects on the rate scale without a class-by-merit interaction.
Apart from constants independent of the parameters, the log-likelihood is
Differentiating gives the Poisson regression margin-matching score equations
Thus the fitted totals equal the observed totals in each class and merit category, including the reference categories by subtraction. In design matrix notation these equations are . Fisher scoring solves them iteratively; the reported three iterations describe the numerical fit, not an additional statistical result.
The regression intercept gives the baseline fitted claim rate per insured car year. The merit rate ratios, relative to merit at the same class, are
The class rate ratios, relative to class at the same merit, are
Each predicted cell rate is the baseline rate times its class and merit multipliers. The output's coefficient-to-standard error ratios are asymptotic normal Wald test statistics under the fitted Poisson regression, despite the software label “t value”; there is no estimated Gaussian residual scale in this model. All nonreference effects have very large absolute ratios.
The null Poisson deviance is on nineteen statistical degrees of freedom; adding the seven factor coefficients reduces it by . But the final Poisson deviance is still on twelve statistical degrees of freedom, and the deviance residuals include values near and . These are far beyond what a well-fitting unit-dispersion model would normally produce. The additive Poisson model fits substantially better than a common rate, but remains grossly inadequate. A class-by-merit interaction, overdispersion, or statistical dependence between claims could contribute; the summary alone does not distinguish these explanations. Its very small nominal standard errors and formal Wald tests rely on the rejected model, so improved mean structure and error assumptions are needed before treating them as reliable uncertainty assessments.
A general exponential family of order has the representation
The canonical sufficient statistics are independent of the parameter, the support is fixed, and the cumulant function normalizes the probability density function. Minimality rules out a redundant affine relation among the sufficient statistics. If ranges over an open subset of , , and is a smooth embedding with derivative of rank , its image is a -dimensional parameter surface in the -dimensional natural parameter space: this is a curved exponential family. In a genuinely curved example the surface is nonaffine.
Take the normal distribution with . Its log density expands as
It is a curved exponential family with sufficient statistics and natural parameters , . The ambient full normal distribution family has , and
The curve is with , and its derivative with respect to has rank one. Thus the mean and variance are constrained by ; the two canonical parameters cannot vary independently. Concavity of an ambient canonical log likelihood need not persist along a curved parameter surface.
For , integrating the exponential tilt gives the moment-generating function
At an interior natural parameter this is finite in a neighbourhood of , so differentiation under the integral sign is justified by the nearby exponential moments. Differentiating its logarithm once and twice at zero gives
Alternatively the score function is ; its mean zero and its covariance matrix give the same exponential-family derivative identities. For any vector , . In a minimal exponential family this variance is strictly positive whenever , proving strict convexity of the cumulant function. For a general exponential family whose sufficient statistic is , these identities describe the mean and covariance matrix of , rather than necessarily those of itself.
In a generalized linear model, the linear predictor is , a linear combination of the covariates with unknown coefficients. The link function relates this predictor to the mean: . The canonical link function chooses the natural parameter of the exponential family, namely . Differentiating the normalization of the density shows that , so the canonical link function is on the admissible parameter interval.
For a Bernoulli distribution, the probability mass function can be written as
Thus and . The canonical link is the logit:
Assume a common support, a differentiable likelihood, and dominated differentiation under the integrals. The score function has mean zero. Its second moment is the Fisher information . If is unbiased, differentiation of gives , hence . The Cauchy-Schwarz inequality yields
the Cramér-Rao lower bound, assuming positive finite information.
Equality holds precisely when the centred estimator and score are linearly dependent almost surely. The covariance fixes the factor, giving . If the same unbiased estimator is efficient for every parameter on a regular interval, integrate this identity in :
Exponentiation gives , an exponential family. This proves efficient unbiased estimator implies exponential family under the stated common-support regularity, rather than asserting it for nonregular support-dependent models.