If has an exponential distribution with rate , a full observation has Fisher information . Recording only for a fixed instead gives a Bernoulli distribution with parameter . The chain rule for its score function gives
To see the strict inequality, write . For the first two terms already exceed . For , the function is positive at one, its derivative is positive there, and that derivative is increasing for . Thus every positive threshold discards information. For independent observations, both information quantities multiply by the sample size. The Cramér-Rao bound and asymptotic normality of a maximum likelihood estimator explain the corresponding loss of attainable precision.
For an independent statistical sample from an Inverse Gaussian distribution with mean and known shape , the maximum-likelihood estimator of is the sample mean. To prove this, expand the log-likelihood function to get . Its score function is , positive below and negative above it. Taking the negative expected second derivative gives one-observation Fisher information . Thus the regular asymptotic normality of a maximum likelihood estimator yields . The same conclusion follows from the central limit theorem and the distribution's variance .
Maximum likelihood estimation Created 2026-09-24 Updated 2026-10-14
Maximum likelihood estimation is a method of statistical estimation: within a specified statistical model, choose the statistical parameter values that maximize the likelihood function of the observed data. For a parameter set and observations , its exact requirement is
The equality requires an attained, finite maximum; a supremum approached only at an excluded boundary is not enough. The resulting rule is a maximum-likelihood estimator. The likelihood function uses a probability mass function for discrete data and a probability density function for continuous data. See MIT 18.05, Reading 10b, sections 3 and 3.2.
The direction of the comparison is the interesting part: hold the observations fixed and vary the statistical parameter. Ten independent Bernoulli trials with eight successes and two failures give the maximum-likelihood estimate , as derived in log-likelihood. This chooses the member of the statistical model that assigns the greatest mass to that outcome. It does not assert an posterior probability that this parameter value is true. Bayesian statistics combines the same likelihood function with a prior distribution to obtain a posterior distribution; maximum a posteriori estimation maximizes that posterior mass or density instead.
The method turns many fitting tasks into one maximization problem. In the normal distribution model with known variance, it recovers the sample mean and ordinary least squares, while binomial regression and Poisson regression use different sampling assumptions to build their own likelihood functions. The log-likelihood turns independent contributions into a sum, exposing score equations and observed information. A peak's location gives an estimate; its shape records how much the observed data distinguish nearby parameter values.
Existence, uniqueness, bias, and statistical consistency are separate questions. Normal mean and variance maximum-likelihood estimators include a variance estimate with bias, likelihood supremum at an excluded boundary gives nonexistence, and the Neyman-Scott incidental parameter problem gives inconsistency. The usual asymptotic normality of a maximum likelihood estimator requires assumptions: a consistent estimator at an interior true parameter in a fixed finite-dimensional model with identifiability, sufficient smoothness and control of log-likelihood derivatives, and finite nonsingular Fisher information. A common parameter-independent support of a probability distribution is part of a standard sufficient framework; see Rigollet, MIT 18.650, Maximum Likelihood Estimation, slides 22-23. This is a description of the regular setting, not a complete theorem's hypotheses. The exact endpoint limit for a shifted exponential distribution shows why changing support can produce a different rate and a nonnormal limit. These properties do not follow just from maximizing a function.
In his 1922 paper in the Royal Society's Philosophical Transactions A, R. A. Fisher explicitly distinguished likelihood from probability and described choosing the parameter values with greatest likelihood. Section 6 also discusses comparing likelihoods after giving the maximum value one, the convention used in the figures here. See Fisher, On the Mathematical Foundations of Theoretical Statistics, section 6, pp. 326-328, DOI 10.1098/rsta.1922.0009.
This one-parameter normal distribution model has positive mean equal to its standard deviation. It is a curved exponential family: both the linear and quadratic natural parameters depend on the same scalar statistical parameter. It combines information about the center and spread, so the sample mean alone does not use all available Fisher information.
For a statistical sample of size , let and . Up to an additive constant the log-likelihood function is . Its score function is . When , the numerator changes from positive to negative at its unique positive root, so the global maximum-likelihood estimator is
The all-zero observation vector has zero probability and is an exceptional case with an unbounded likelihood function near zero. Since and , the expected Fisher information is . Thus regular asymptotic normality of a maximum likelihood estimator gives . The optimal asymptotic combination of a normal sample mean and standard deviation attains the same first-order precision.
Here the negative binomial distribution has mean parameter and size parameter . Its support in the original PDF is all nonnegative integers; a finite endpoint appearing in the TeX aid would be incorrect. The probability generating function, obtained from the generalized binomial series, is
Differentiating at one gives and . Since the variance is , this proves
The extra positive term explains its overdispersion relative to a Poisson distribution.
For one observation, write the log-likelihood as
The mean score function and its cross derivative are
Taking expectations gives . In addition,
For reference, with the digamma function, the size score function is
Its variance is finite and positive for finite positive . Thus negative binomial mean-size parameter orthogonality makes the expected Fisher information matrix diagonal. For an independent identically distributed sample and a regular identifiable interior true parameter, asymptotic normality of a maximum likelihood estimator gives
Hence the asymptotic correlation is zero. This is first-order asymptotic independence, not a claim of exact finite-sample independence. Indeed the mean score yields at an interior fit, while the size fit also depends on the observed dispersion. The argument excludes a true boundary such as the Poisson limit .
Write for the scalar intercept denoted in the question, to distinguish it from the individual means . The log-likelihood of the independent Poisson observations is
Differentiating gives the score equations for an interior maximum-likelihood estimator:
In particular, when ,
The left side is a weighted covariate mean whose derivative is its weighted variance, so it is strictly increasing when the covariates are not all equal. An interior solution is unique if the observed count-weighted mean is strictly between the smallest and largest covariates. At an extreme it may require a limiting slope, and if all counts are zero the maximum occurs as , rather than at finite parameters. Constant covariates make the slope and intercept unidentifiable.
Put . The negative Hessian matrix is deterministic at given parameters and is also the Fisher information matrix:
Under the usual interior, increasing-information regularity for asymptotic normality of a maximum likelihood estimator, replace the unknown means by their fitted values and read the lower-right entry:
Equivalently, Poisson slope information after eliminating an intercept is , with . This explains both the positive denominator for varying covariates and the loss of slope information when their spread is small.
Write and for the individual and total score functions. Under the usual regularity assumptions, and , the nonsingular per-observation Fisher information matrix. The central limit theorem and weak law of large numbers give
Rearranging the stipulated score expansion, including its remainder, gives
Slutsky theorem therefore proves the asymptotic normality of a maximum likelihood estimator:
A consistent plug-in Fisher information matrix yields the Wald statistic . Under the null hypothesis, , so a test of asymptotic size rejects when exceeds the quantile of that chi-squared distribution. A consistent observed Fisher information matrix gives the same limiting test.
With an -dimensional parameter of interest and nuisance parameter , partition accordingly. The marginal asymptotic covariance matrix of is the block of , whose inverse is the efficient information
Thus the nuisance-adjusted Wald statistic is , with limiting null distribution . Using alone is generally incorrect when the nuisance parameter is estimated; it is valid when the cross-information vanishes.
For the Inverse Gaussian distribution in this problem, expand the exponent to obtain the log likelihood, up to terms independent of both parameters,
Its score functions are
For each positive , the first score changes sign from positive to negative at , so this is the global maximizing mean parameter of an exponential family. Substituting it in the second score gives
The reciprocal function is strictly convex, so the denominator is positive for a nonconstant sample. The profile log likelihood is strictly concave in and tends to minus infinity at both endpoints, proving this is a maximum. Under continuous model sampling, a sample of size at least two is nonconstant almost surely. For a constant sample, including a one-observation sample, no finite joint maximum-likelihood estimator exists: the likelihood increases without bound as at .
For one observation, differentiating again and using gives
This parameter orthogonality proves that estimating does not reduce the efficient information for . The Wald statistic has the same algebraic form in the two cases:
Using information evaluated at instead is another first-order equivalent convention.
The printed claim of coincidence needs a first-order interpretation. Known and unknown need not give numerically identical Wald statistics, because their maximum-likelihood estimators differ. With known, direct maximization gives
For example, observations and known give and . At , the displayed Wald convention gives and , respectively. Thus exact equality is false. Under the null hypothesis, however, , hence . Their centered, standardized estimates differ by , so their Wald statistics differ by . This is the valid asymptotic coincidence supplied by inverse Gaussian shape estimation and nuisance orthogonality.
Put and . The integrable absolute-log envelope makes these expectations finite. Compactness and continuity of the likelihood ensure that a maximum exists, and maximizing it is equivalent almost surely to maximizing .
First identify the unique maximizer of . Let on the support of the true density. Since ,
Equality requires almost surely under the true distribution and no mass outside its support, so the two probability distributions coincide. By identifiability, this happens only for . Equivalently is the Kullback-Leibler divergence from the true distribution to the candidate.
Fix and let . If this set is empty there is nothing to prove. Otherwise it is compact, so continuity and the strict maximum imply
On the event , every point of has , whereas . Hence no maximizer can belong to . The supplied uniform law of large numbers therefore proves compact-parameter consistency of maximum likelihood:
Under the usual regularity conditions, with nonsingular one-observation Fisher information matrix , the asymptotic normality of a maximum likelihood estimator is
where and is the score function.
For the uniform model, let . For a sample in its support, the likelihood is , decreasing on its nonzero region. Thus the uniform endpoint maximum-likelihood estimator is
The all-zero sample is a probability-zero exception and may be assigned an arbitrary positive estimate. Independence gives on . For ,
for this probability is zero. This proves consistency directly.
Let . Its exact cumulative distribution function is
For fixed , this converges to , so . In particular the errors are of order . Explicitly, for every fixed and sufficiently large ,
Consequently
One violated regularity condition is parameter-independent support: the density is supported on . Differentiation under the density integral is not valid in the usual regular-model form. Indeed the interior score equals , whose expectation is not zero; the changing endpoint contributes the omitted boundary term. Thus the uniform model is handled by its exact maximum distribution, rather than by the regular asymptotic normality theorem.
Work at a regular admissible parameter value with , so the score identities in the question apply. For one observation from the exponential dispersion family, the score function and second derivative are
The zero expected score immediately gives . The equality between expected negative curvature and score variance then gives
Thus
The admissible density parameters necessarily make this variance nonnegative; the algebra does not require an unstated sign assumption beyond the regular family and .
With independent observations and canonical parameters , the chain rule gives the vector score function
A finite interior maximum-likelihood estimator satisfies its score equation, proving
Differentiating again, using , gives
This Hessian is independent of the observations, so taking expectation gives exactly the requested Fisher information matrix. In particular, is not the inverse observation-variance matrix: the linear predictor here is the canonical parameter, not the mean.
For the two-group Poisson regression, , , and . Put , , using here for the second group's mean. There are observations, and
Hence the total Fisher information matrix and its inverse are
The theorem on asymptotic normality of a maximum likelihood estimator says that in a regular identifiable model with nonsingular information the estimator is approximately multivariate normal, centred at the true parameter and with covariance equal to the inverse total information. Here one can regard one observation from each group as an independent pair and apply its usual iid version to such pairs. Therefore
For a direct check, the second score equation gives the fitted second-group mean , and the first then gives the fitted first-group mean . Thus the Poisson log-rate ratio estimate is . The delta method, using independent sample-mean variances and , gives the same variance . If a group total is zero, its finite log-mean estimator fails to exist; with fixed positive true means this event has probability tending to zero exponentially as grows.
The likelihood function for the shifted exponential distribution is
It increases strictly up to the smallest observation and is zero beyond it. Thus shifted exponential maximum likelihood gives
For , independence and the survival function of the exponential distribution give the exact order statistic law
In particular, almost surely and . This proves statistical consistency. The exact endpoint limit for a shifted exponential distribution is stronger than an asymptotic approximation:
The limit is one-sided and nonnormal; the moving support endpoint explains why a regular square-root- asymptotic normality of a maximum likelihood estimator theorem does not apply here.
In the species-only model, the log odds for adult species A is the intercept, so with standard error 0.07824. This also follows directly by pooling its vials: there are 570 other-species eggs among 799 eggs eaten, giving and
For a regular maximum-likelihood estimator, asymptotic normality of a maximum likelihood estimator gives , with information . Substitution of gives . An approximate 95% Wald confidence interval is
This interval treats the selected species-only model as fixed; it does not account separately for uncertainty from choosing that model.
For independent identically distributed observations , the likelihood function and log-likelihood are
A maximum-likelihood estimator is a measurable choice , when a maximizer exists. It need not be unique. The one-observation score function is , and the Fisher information matrix is
Under the regularity assumptions, differentiation beneath the integral gives . Differentiating again gives the second information identity. Independent observations have total Fisher information .
Assume fixed parameter dimension, an interior true parameter , a positive-definite matrix , the standard differentiability and integrability conditions, and statistical consistency of . Then the asymptotic normality of a maximum likelihood estimator is
Here is the information per observation. Thus the leading covariance matrix of the estimator itself is .
To prove this, statistical consistency places the estimator in an interior ball about with probability tending to one, where its score function vanishes. Write and use the integral first-order Taylor formula for a vector map:
This integral matrix is needed in a vector problem; one does not have to assert a common scalar mean-value point for every score component. Put
The regular local uniform law of large numbers, continuity of the expected Hessian matrix, and statistical consistency give . The multivariate central limit theorem gives
The limiting information matrix is nonsingular, so . Solving the Taylor identity and applying the Slutsky theorem proves the displayed normal limit, since . Boundary parameters or singular information are outside this regular theorem.
Let be the parameter-contrast vector for the specified game: it has entries for , entries for , and zero elsewhere. Put
Under the stated regularity assumptions, asymptotic normality of a maximum likelihood estimator gives
where . Since , the delta method gives
Equivalently, for large ,
with .
Under the usual regularity conditions, consistency and asymptotic normality of a maximum likelihood estimator give
where is the one-observation Fisher information matrix. In a scalar model this reads .
For a fixed true parameter , the usual asymptotic normality of maximum likelihood estimators gives
Equivalently, for a large sample, is approximately normally distributed with mean and variance . Replacing the unknown variance by its consistent plug-in estimate gives the Wald confidence interval
Here is a standard-normal quantile. Intersect with the physical parameter range if needed. This is an interior, large-sample approximation; the degenerate estimate from does not justify a zero-width finite-sample interval.
For an interior true parameter and large , standard asymptotic normality of a maximum likelihood estimator gives
Estimate its standard error by . A two-sided approximate Wald confidence interval is , intersected with the parameter range . The normal quantile is chosen so a standard normal lies between its positive and negative values with probability .
For the independent statistical sample, write the likelihood function and log-likelihood function as
The expected Fisher information in the whole statistical sample is
Under the regularity conditions that permit differentiation under the integral sign, the score function has expected value zero and
For an interior statistical parameter, assuming identifiability and , the regular asymptotic normality of a maximum likelihood estimator states
Thus the large-sample normal approximation is . The inverse information is the leading sampling variance approximation; the displayed convergence specifies its precise asymptotic meaning.
Put , and . The normal distribution model gives
and, up to a constant independent of , its log-likelihood function is
The score function and its zero equation are
For , this quadratic equation has exactly one positive root, giving the maximum-likelihood estimator
The numerator of the score function is positive between its negative and positive roots and negative above its positive root. Thus the likelihood function increases up to and decreases afterwards: this is its unique global maximum on . Also, at both ends. The all-zero observation vector has probability zero; on that exceptional vector and there is no attained maximum.
The second derivative and the relevant moments are
Therefore the expected Fisher information is
The support of a probability distribution is the whole real line, independent of the positive statistical parameter. The law of large numbers gives and in probability, so the displayed maximum-likelihood estimator is a consistent estimator. The regular asymptotic normality of a maximum likelihood estimator now yields
Thus the requested leading asymptotic variance on the unscaled estimator's scale is . The normal mean proportional to standard deviation model provides a reusable version of this calculation.
Write , with the natural logarithm. The likelihood function and log-likelihood function are
The score function is and the second derivative is . Thus the log-likelihood function is strictly concave, its stationary point is the unique global maximum, and the maximum-likelihood estimator is
The one-observation Fisher information is . The support of a probability distribution here is independent of , so the regular asymptotic normality of a maximum likelihood estimator gives
There is also a direct check: has an exponential distribution with rate , since for . Its mean and variance are and . Apply the central limit theorem to and the delta method to to obtain the same result.
Expanding the exponent, the log-likelihood function is
where is independent of . Its score function and second derivative are
The first is positive below and negative above it, so the unique maximum-likelihood estimator is
Using , the one-observation Fisher information is . The support does not depend on the interior parameter , and the regular asymptotic normality of a maximum likelihood estimator gives
This is Inverse Gaussian mean estimation with known shape one. Equivalently, the Inverse Gaussian distribution has variance , so the same result follows directly from the central limit theorem applied to its sample mean.