Maximum likelihood estimation

Maximum likelihood estimation is a method of statistical estimation: within a specified statistical model, choose the statistical parameter values that maximize the likelihood function of the observed data. For a parameter set $\Theta$ and observations $x$, its exact requirement isThe equality requires an attained, finite maximum; a supremum approached only at an excluded boundary is not enough. The resulting rule is a maximum-likelihood estimator. The likelihood function uses a probability mass function for discrete data and a probability density function for continuous data. See MIT 18.05, Reading 10b, sections 3 and 3.2.

The direction of the comparison is the interesting part: hold the observations fixed and vary the statistical parameter. Ten independent Bernoulli trials with eight successes and two failures give the maximum-likelihood estimate $\hat{p}=0.8$, as derived in log-likelihood. This chooses the member of the statistical model that assigns the greatest mass to that outcome. It does not assert an $80\%$ posterior probability that this parameter value is true. Bayesian statistics combines the same likelihood function with a prior distribution to obtain a posterior distribution; maximum a posteriori estimation maximizes that posterior mass or density instead.

The method turns many fitting tasks into one maximization problem. In the normal distribution model with known variance, it recovers the sample mean and ordinary least squares, while binomial regression and Poisson regression use different sampling assumptions to build their own likelihood functions. The log-likelihood turns independent contributions into a sum, exposing score equations and observed information. A peak's location gives an estimate; its shape records how much the observed data distinguish nearby parameter values.

Existence, uniqueness, bias, and statistical consistency are separate questions. Normal mean and variance maximum-likelihood estimators include a variance estimate with bias, likelihood supremum at an excluded boundary gives nonexistence, and the Neyman-Scott incidental parameter problem gives inconsistency. The usual asymptotic normality of a maximum likelihood estimator requires assumptions: a consistent estimator at an interior true parameter in a fixed finite-dimensional model with identifiability, sufficient smoothness and control of log-likelihood derivatives, and finite nonsingular Fisher information. A common parameter-independent support of a probability distribution is part of a standard sufficient framework; see Rigollet, MIT 18.650, Maximum Likelihood Estimation, slides 22-23. This is a description of the regular setting, not a complete theorem's hypotheses. The exact endpoint limit for a shifted exponential distribution shows why changing support can produce a different rate and a nonnormal limit. These properties do not follow just from maximizing a function.

In his 1922 paper in the Royal Society's Philosophical Transactions A, R. A. Fisher explicitly distinguished likelihood from probability and described choosing the parameter values with greatest likelihood. Section 6 also discusses comparing likelihoods after giving the maximum value one, the convention used in the figures here. See Fisher, On the Mathematical Foundations of Theoretical Statistics, section 6, pp. 326-328, DOI 10.1098/rsta.1922.0009.

For independent normal distribution observations with mean $\mu >0$ and variance ${\mu }^2$, put $S_1={\sum }_i{X}_i$, $S_2={\sum }_i{X}_i^2$. The Fisher-Neyman factorization theorem makes $(S_1,S_2)$ a sufficient statistic, since the likelihood is proportional to ${\mu }^{-n}\exp [-S_2/(2{\mu }^2)+S_1/\mu ]$. Its logarithmic derivative has numerator $S_2-S_1\mu -n{\mu }^2$. When $S_2>0$, this changes sign once on the positive half-line and gives the displayed global maximum-likelihood estimator. The zero sample is a probability-zero exception with unbounded likelihood as $\mu \downarrow 0$.

For independent $p$-variate multivariate normal observations with unknown mean and unrestricted positive-definite covariance matrix, the centered scatter has rank at most $n-1$. If it is singular, choose the mean to be the sample mean and shrink the covariance on a null direction of the scatter. The quadratic likelihood term stays fixed while the determinant tends to zero, so the density likelihood tends to infinity. Thus a nonsingular covariance maximum likelihood estimate exists only when the centered scatter is positive definite; for a nonsingular Gaussian population this occurs almost surely exactly when $n>p$.

On a compact parameter space, suppose the expected log-likelihood is continuous, uniquely maximized at the true parameter, and the normalized empirical log-likelihood converges uniformly in probability to its expectation. On the compact complement of every neighbourhood of the truth, the expected criterion has a strictly positive gap below its maximum. Uniform approximation by less than one-third of this gap excludes every empirical maximizer from that complement. Identifiability and nonnegativity of Kullback-Leibler divergence supply the unique expected maximizer. The proof works for any measurable choice of maximizer.

A shifted exponential distribution has location estimate equal to the sample minimum and rate estimate given above, for a nondegenerate sample of at least two observations. The minimum exceeds the true location on average by $1/(n\lambda )$. The residual sum above the minimum is gamma with shape $n-1$, so the rate estimate has expectation $n\lambda /(n-2)$ for $n>2$ and infinite expectation at $n=2$. The estimated location plus reciprocal rate is the unbiased sample mean.

For independent observations $Y_i=\theta +E_i$ with $E_i$ having exponential distribution of known rate $\lambda >0$, the likelihood increases in $\theta$ until the smallest observation, so the maximum-likelihood estimator is $Y_{(1)}$. Its survival function is $\mathbb{P}(Y_{(1)}-\theta >t)=e^{-n\lambda t}$ for $t\ge 0$. Therefore $n\lambda (Y_{(1)}-\theta )$ has exactly the unit-rate exponential distribution for every sample size. This gives statistical consistency and a one-sided nonnormal limit with rate $n$, illustrating how a support endpoint changes usual regular likelihood asymptotics.

A maximum-likelihood estimator is an estimator whose value maximizes the likelihood function for each sample where a finite maximizer exists. Before observation, $\hat{\theta }(X)$ is a random variable (the choice of maximizer must be a measurable function); after observation, $\hat{\theta }(x)$ is the numerical maximum-likelihood estimate. A formula must specify the allowed parameter set, including whether its endpoints belong to it.

For ten independent Bernoulli trials, write $K={\sum }_{i=1}^{10}{X}_i$. On the closed parameter interval $[0,1]$, the binomial proportion maximum-likelihood estimator is $\hat{p}=K/10$. Eight successes give the estimate $0.8$. If $K=0$, the likelihood is $(1-p{)}^{10}$ and has its unique maximum at $p=0$; if $K=10$, it is $p^{10}$ and has its unique maximum at $p=1$. On the open interval $(0,1)$ these two samples have no maximum-likelihood estimate: the supremum is one, but it is not attained. This is the same issue as likelihood supremum at an excluded boundary.

A complete continuous example uses independent and identically distributed random variables $X_i\sim N(\mu ,1)$ with unknown mean $\mu \in \mathbb{R}$ and known standard deviation one. Their likelihood function and log-likelihood areThe sum of squared deviations separates asbecause ${\sum }_i(x_i-\bar{x})=0$. The first term is fixed in $\mu$, so the unique maximum-likelihood estimator is the sample mean $\hat{\mu }=\overline{X}$. This also proves the equivalence with ordinary least squares for this model. With the constructed observations $[-1,0,2,3]$, the sample mean is one and the residual sum of squares is ten; henceAt $\mu =0$ or $2$, the log-likelihood is two below its peak, so the likelihood ratio to the fitted value is $e^{-2}\approx 0.135335$. The absolute likelihood at the peak is $(2\pi {)}^{-2}{e}^{-5}\approx 0.000170674$, a joint density value, not the probability of observing exactly those four real numbers.

The following figure compares that sample with a constructed sixteen-observation sample consisting of four repetitions of the displayed four values, modeled as sixteen independent draws. Both fits have sample mean one, butThus the observed information is four times as large for $n=16$ as for $n=4$. The distance from the peak at relative height $e^{-2}$ decreases from one to $1/2$: the width scales as $1/\sqrt{n}$. This is a comparison of two stipulated samples; merely copying recorded observations would not create new independent evidence.

Even uniqueness needs checking. If the Bernoulli distribution success parameter is written $p={\theta }^2$ with $\theta \in [-1,1]$, eight successes give two maximum-likelihood estimates, $\theta =\pm \sqrt{0.8}$, because both give the same probability distribution. The problem is a failure of identifiability. The special cases below show other ways an MLE can have bias or fail the usual asymptotic normality.

A likelihood need not attain its supremum when the parameter space excludes a boundary point. For example, with parameter $0<\theta \le 1$ and a zero Poisson count, the likelihood $e^{-\theta }$ is strictly decreasing and has no maximum. Extending the parameter space to include zero gives an extended maximum-likelihood estimate there. A proposed estimator formula involving that boundary value must distinguish the extended estimate from an actual maximizer in the original space.

For independent $U[0,\theta ]$ observations, the likelihood decreases as ${\theta }^{-n}$ on $[{\max }_i{Y}_i,\infty )$, so its maximizer is the sample maximum almost surely. Its exact distribution is ${\mathbb{P}}_{\theta }({\hat{\theta }}_n\le y)=(y/\theta {)}^n$ for $0\le y\le \theta$. This proves consistency and an endpoint error of order $1/n$, with the displayed exponential limit. Parameter-dependent support violates the usual regularity condition behind nondegenerate square-root-sample-size normal limits; the ordinary interior score is not centered.

A growing number of nuisance parameters can prevent a maximum-likelihood estimator of a common parameter from being consistent. With two independent normal observations of mean ${\mu }_i$ per group, the profiled variance estimate ${\sum }_i(X_{1i}-X_{2i}{)}^2/(4n)$ tends to half the true variance, because each unknown mean is estimated using only two observations.

For density $r(x-\theta {)}^{r-1}$ on $[\theta ,\theta +1]$, $0<r<1$, the interior likelihood function increases without bound as $\theta$ approaches the sample minimum. The extended-likelihood estimator is that minimum. An arbitrary finite reassignment of the density at the endpoint can eliminate an attained finite maximum, so the endpoint convention must be stated.

For independent observations from the exponential distribution with rate $\theta$, the log-likelihood is $n\log \theta -\theta {\sum }_i{X}_i$, soThe central limit theorem and delta method give

A maximum-likelihood fitted value is a response mean or probability evaluated at a maximum-likelihood estimate. In a parametric mean model ${\mu }_i(\theta )$ it is ${\hat{\mu }}_i={\mu }_i(\hat{\theta })$.

For independent and identically distributed random variables $X_1,\dots ,X_n\sim N(\mu ,{\sigma }^2)$, the maximum-likelihood estimators areThe first is unbiased, while $\mathbb{E}{\hat{\sigma }}^2=(n-1){\sigma }^2/n$.

If $\hat{\theta }$ maximizes a likelihood and the target parameter is $\eta =g(\theta )$, a maximum-likelihood estimate of $\eta$ is $g(\hat{\theta })$, with the usual interpretation when $g$ is not one-to-one.

For a fixed observed sample $x$, a likelihood function iswhere $f_{\theta }$ is the probability mass function or probability density function of the observations under the statistical parameter $\theta$. All densities must be defined relative to the same parameter-independent dominating measure. The sample is fixed; the parameter is the function's argument. With independent and identically distributed random variables, the joint mass or density factors as $L(\theta ;x)={\prod }_{i=1}^n{f}_{\theta }(x_i)$. This definition and the distinction between sampling probability and parameter uncertainty are explained in MIT 18.05, Reading 10b, sections 3 and 3.2.

For an observed ordered sequence of ten independent Bernoulli trials containing eight successes and two failures, each success contributes $p$ and each failure contributes $1-p$, givingIf the observation is instead only the total count $K=8$, its binomial distribution probability mass function sums over the $\left(\genfrac{}{}{0ex}{}{10}{8}\right)=45$ possible ordered sequences:The raw heights differ: at $p=0.8$ they are approximately $0.0067108864$ and $0.301989888$. Their maximum-likelihood estimates are identical. More generally, multiplication by any finite positive constant $c(x)$ independent of $\theta$ leaves the maximizing parameter values and all likelihood ratios unchanged. Multiplication by a parameter-dependent factor can change the MLE; dropping such a factor is not justified. Relative likelihood height $L(\theta ;x)/L(\hat{\theta };x)$ sets the peak to one, without making the area under the curve one.

Normalization of $f_{\theta }$ is over possible data values for each fixed parameter. It imposes no requirement that ${\int }_{\Theta }L(\theta ;x)d\theta =1$. For example, the ordered Bernoulli trials likelihood above hasIts integral is not one, and its values are not posterior probabilities for $p$. When the Bayesian posterior is defined, Bayes' theorem multiplies the likelihood by a prior density and normalizes that product. Choosing to normalize a likelihood function alone is an additional choice, such as using a uniform prior in those parameter coordinates, rather than part of the definition of likelihood.

For observations with a probability density function, the probability of an exact sample point is zero. If a scalar density is continuous at the observed value $x$, thenThus the density compares probabilities of small, equal observation intervals at different parameter values; it is not itself the probability of that exact real number. The equal-width factor is independent of the parameter. This gives visual intuition for density likelihoods, as in MIT 18.05, Reading 10b, section 4. The normal distribution example in maximum-likelihood estimator makes the comparison explicit. If only rounded intervals are recorded, their integrated probabilities give the exact observation likelihood; using midpoint densities is an approximation.

The log-likelihood is the natural logarithm of the likelihood function:For positive finite likelihood, the natural logarithm is a strictly increasing function, so the maximizing parameter values are unchanged. Set $\ell =-\infty$ where $L=0$; the same ordering then includes zero-likelihood boundaries whenever a positive finite maximum exists. For independent and identically distributed random variables, a product becomes a sum,This makes derivatives simpler and avoids forming products of tiny numbers in floating-point arithmetic. These equal-maximizer and additive formulas underlie the worked calculations in MIT 18.05, Reading 10b, section 3.1.

For the ordered sample of eight successes and two failures in likelihood function, on $0<p<1$,The score equation ${\ell }^{\prime }(p)=0$ gives $8(1-p)=2p$, hence $\hat{p}=8/10=0.8$. The negative second derivative proves strict concavity, so this is the unique interior maximum. Both counts are positive, so $L(0)=L(1)=0$, while $L(p)>0$ inside: the same point is the unique global maximum on $[0,1]$. Counting only successes adds $\log 45$ to the log-likelihood, leaving both derivatives and the MLE unchanged. For all-success or all-failure samples, the boundary analysis in maximum-likelihood estimator applies instead of an interior score equation.

The two panels below compare $k=2,5,8$ successes out of ten Bernoulli trials. Their peaks are at $\hat{p}=k/10$. The left panel uses relative height; the right panel uses the log-likelihood difference from each peak. These are the same comparison on two scales:A difference of $-2$ means relative height $e^{-2}$, and a difference of $-10$ means $e^{-10}$; the log-likelihood scale makes small relative heights easier to see. Neither vertical axis is a parameter probability density function. The zero likelihood endpoints have log-likelihood $-\infty$ and are outside the finite vertical range in the right panel.

Multiplying a likelihood function by a positive parameter-independent constant adds a constant to its log-likelihood. Subtracting its peak removes that constant. This is why the ordered-data and count-data curves coincide in this figure after rescaling. When differentiable, the gradient of the log-likelihood is the score function; the negative Hessian matrix is the observed information. A solution of a score equation still needs checking against other stationary points and boundaries: maximizing the log-likelihood does not automatically make it concave.

If a statistic T has marginal density depending on a parameter of interest but not on a nuisance parameter, that density gives a likelihood for the interest parameter. It is obtained by integrating out other observed coordinates. In contrast, profile likelihood maximizes the full likelihood over the nuisance parameter. This sampling-statistic construction is different from Bayesian model evidence, which integrates parameters against a prior. No prior is required for a nuisance-free sampling marginal.

A conditional likelihood evaluates the conditional distribution of the observed data given a specified statistic. If that distribution does not involve a nuisance parameter, it permits inference about the remaining parameter without estimating the nuisance parameter. The conditioning statistic need not be ancillary; possible information loss must be assessed within the model. Conditional maximum likelihood maximizes this function.

For a regular two-parameter exponential family, condition on the sample mean of the nuisance canonical statistic. Exponential tilting to its observed value and the saddlepoint density approximation give a nuisance-statistic density proportional to $d_{\lambda \lambda }^{-1/2}$ times its large-deviation exponential. Dividing the joint likelihood by this density gives the profile log-likelihood plus the displayed adjustment, up to a data-only constant. The sign is positive. This construction assumes an interior saddle with positive curvature and a sufficiently regular nonlattice density; conditioning on a lattice statistic requires the corresponding mass approximation.

An observed-data likelihood integrates the full-data statistical probability density over unobserved values, rather than inserting arbitrary values or discarding incomplete records. When the observation pattern is random, its mechanism belongs to the joint observation model unless the conditions for an ignorable missingness mechanism permit its factor to be omitted.

A Bayesian deviance is minus twice the log-likelihood, with any chosen additive data-only constant held consistent across the models being compared. Its Bayesian posterior expectation measures average fit. In the deviance information criterion, evaluating it at a parameter's posterior mean also enters the effective complexity penalty. This likelihood-based convention differs by a data-only constant from a saturated-model exponential-family deviance when a common saturated model exists.

A conditional likelihood function conditions on an initial observation or another specified statistic. In an autoregressive model, conditioning on the first observed state avoids modeling its initial density. For Gaussian transitions, maximizing this likelihood is a regression problem; treating an autoregressive coefficient as known changes the exact distribution of an estimated mean.

For a parameter of interest $\psi$ and nuisance parameter $\lambda$, the profile likelihood isIt compares values of $\psi$ after fitting the nuisance parameter as favourably as possible at each value.

For parameter of interest $\psi$, constrained nuisance fit ${\hat{\lambda }}_{\psi }$, and suitable ancillary sample coordinates $(\hat{\psi },\hat{\lambda },a)$, this adjustment includes both nuisance observed information and a sample-space Jacobian determinant. The derivative holds $\hat{\psi },a$ fixed. Implicit differentiation of the constrained nuisance score function gives the equivalent expression ${\ell }_p+\frac{1}{2}\log |j_{\lambda \lambda }|-\log |{\ell }_{\lambda ;\hat{\lambda }}|$. The semicolon distinguishes data-coordinate differentiation from parameter differentiation. Omitting the Jacobian requires an additional justified simplification; the information adjustment alone is not universal.

For independent Inverse Gaussian distributions with shape $\psi$ and mean $\lambda$, the constrained and full nuisance maximum-likelihood estimates both equal $\bar{Y}$, giving a unit sample-coordinate Jacobian determinant. The nuisance observed information there is $n\psi /{\bar{Y}}^3$. Hence subtracting half its logarithm from the profile log-likelihood replaces the coefficient $n/2$ of $\log \psi$ by $(n-1)/2$. For a nonconstant sample the modified fit is $(n-1)/({\sum }_i{Y}_i^{-1}-n/\bar{Y})$.

For independent exponential distributions with means $\lambda e^{\psi x_i}$ and a nonconstant fixed design, put $A(\psi )={\sum }_i{Y}_i{e}^{-\psi x_i}$. The constrained maximum-likelihood estimate is ${\tilde{\lambda }}_{\psi }=A(\psi )/n$ and the profile log-likelihood is ${\ell }_p(\psi )=-n\log [A(\psi )/n]-\psi {\sum }_i{x}_i-n$. Use the ancillary statistic $a_i=\log Y_i-\log \hat{\lambda }-\hat{\psi }{x}_i$ to specify sample coordinates. At fixed $\hat{\psi },a$, the Jacobian determinant $\partial {\tilde{\lambda }}_{\psi }/\partial \hat{\lambda }={\tilde{\lambda }}_{\psi }/\hat{\lambda }$, while $j_{\lambda \lambda }(\psi ,{\tilde{\lambda }}_{\psi })=n/{\tilde{\lambda }}_{\psi }^2$. Therefore the modified profile likelihood satisfiesThe adjustment is independent of $\psi$. Keeping only the observed information adjustment would miss this cancellation.

For independent $X\sim \mathrm{Gamma}(m,\psi \lambda )$ and $Y\sim \mathrm{Gamma}(n,\lambda )$ in shape-rate notation, $T=X/Y$ has density proportional to ${\psi }^m{t}^{m-1}(1+\psi t{)}^{-(m+n)}$. Its distribution is nuisance-free in lambda. The nuisance maximum is $(m+n)/(\psi X+Y)$, so substituting into the full likelihood gives the same interest-parameter log-likelihood up to a parameter-free term. Equality here is special to the common-scale structure, not a general equality of profiling and marginalization.

The logarithm of the profile likelihood is the full log-likelihood evaluated at the nuisance fit constrained to the specified parameter of interest. It preserves likelihood ratios and maximizers, but is not generally a normalized sampling log-density. Conditioning and integrating out nuisance parameters are different operations.

For $X\sim \mathrm{Bin}(n,\theta )$, maximizing ${\theta }^X(1-\theta {)}^{n-X}$ gives

For independent exponential observations with rate $\theta$, the MLE is $1/\overline{X}$ and has asymptotic variance ${\theta }^2/n$.

In a regular $p$-parameter model with one-observation Fisher information $I({\theta }_0)$,