Log-likelihood Created 2026-09-24 Updated 2026-10-14
The log-likelihood is the natural logarithm of the likelihood function:
For positive finite likelihood, the natural logarithm is a strictly increasing function, so the maximizing parameter values are unchanged. Set where ; the same ordering then includes zero-likelihood boundaries whenever a positive finite maximum exists. For independent and identically distributed random variables, a product becomes a sum,
This makes derivatives simpler and avoids forming products of tiny numbers in floating-point arithmetic. These equal-maximizer and additive formulas underlie the worked calculations in MIT 18.05, Reading 10b, section 3.1.
For the ordered sample of eight successes and two failures in likelihood function, on ,
The score equation gives , hence . The negative second derivative proves strict concavity, so this is the unique interior maximum. Both counts are positive, so , while inside: the same point is the unique global maximum on . Counting only successes adds to the log-likelihood, leaving both derivatives and the MLE unchanged. For all-success or all-failure samples, the boundary analysis in maximum-likelihood estimator applies instead of an interior score equation.
The two panels below compare successes out of ten Bernoulli trials. Their peaks are at . The left panel uses relative height; the right panel uses the log-likelihood difference from each peak. These are the same comparison on two scales:
A difference of means relative height , and a difference of means ; the log-likelihood scale makes small relative heights easier to see. Neither vertical axis is a parameter probability density function. The zero likelihood endpoints have log-likelihood and are outside the finite vertical range in the right panel.
Multiplying a likelihood function by a positive parameter-independent constant adds a constant to its log-likelihood. Subtracting its peak removes that constant. This is why the ordered-data and count-data curves coincide in this figure after rescaling. When differentiable, the gradient of the log-likelihood is the score function; the negative Hessian matrix is the observed information. A solution of a score equation still needs checking against other stationary points and boundaries: maximizing the log-likelihood does not automatically make it concave.
Maximum likelihood estimation Created 2026-09-24 Updated 2026-10-14
Maximum likelihood estimation is a method of statistical estimation: within a specified statistical model, choose the statistical parameter values that maximize the likelihood function of the observed data. For a parameter set and observations , its exact requirement is
The equality requires an attained, finite maximum; a supremum approached only at an excluded boundary is not enough. The resulting rule is a maximum-likelihood estimator. The likelihood function uses a probability mass function for discrete data and a probability density function for continuous data. See MIT 18.05, Reading 10b, sections 3 and 3.2.
The direction of the comparison is the interesting part: hold the observations fixed and vary the statistical parameter. Ten independent Bernoulli trials with eight successes and two failures give the maximum-likelihood estimate , as derived in log-likelihood. This chooses the member of the statistical model that assigns the greatest mass to that outcome. It does not assert an posterior probability that this parameter value is true. Bayesian statistics combines the same likelihood function with a prior distribution to obtain a posterior distribution; maximum a posteriori estimation maximizes that posterior mass or density instead.
The method turns many fitting tasks into one maximization problem. In the normal distribution model with known variance, it recovers the sample mean and ordinary least squares, while binomial regression and Poisson regression use different sampling assumptions to build their own likelihood functions. The log-likelihood turns independent contributions into a sum, exposing score equations and observed information. A peak's location gives an estimate; its shape records how much the observed data distinguish nearby parameter values.
Existence, uniqueness, bias, and statistical consistency are separate questions. Normal mean and variance maximum-likelihood estimators include a variance estimate with bias, likelihood supremum at an excluded boundary gives nonexistence, and the Neyman-Scott incidental parameter problem gives inconsistency. The usual asymptotic normality of a maximum likelihood estimator requires assumptions: a consistent estimator at an interior true parameter in a fixed finite-dimensional model with identifiability, sufficient smoothness and control of log-likelihood derivatives, and finite nonsingular Fisher information. A common parameter-independent support of a probability distribution is part of a standard sufficient framework; see Rigollet, MIT 18.650, Maximum Likelihood Estimation, slides 22-23. This is a description of the regular setting, not a complete theorem's hypotheses. The exact endpoint limit for a shifted exponential distribution shows why changing support can produce a different rate and a nonnormal limit. These properties do not follow just from maximizing a function.
In his 1922 paper in the Royal Society's Philosophical Transactions A, R. A. Fisher explicitly distinguished likelihood from probability and described choosing the parameter values with greatest likelihood. Section 6 also discusses comparing likelihoods after giving the maximum value one, the convention used in the figures here. See Fisher, On the Mathematical Foundations of Theoretical Statistics, section 6, pp. 326-328, DOI 10.1098/rsta.1922.0009.
The P-star approximation describes the sampling density of a regular maximum-likelihood estimator in sample coordinates consisting of the fitted parameter and a suitable ancillary statistic . For a -dimensional parameter, write for the log-likelihood expressed in these coordinates, and for the fitted observed information. The approximation is
The determinant factor supplies the local volume scale, while the exponential retains the full likelihood ratio, rather than replacing it by a quadratic. The normalizing constant is chosen by integrating over the fitted-parameter coordinates; its leading regular large-sample value is . These ingredients arise from a statistical saddlepoint approximation. Their use requires an interior, locally unique fit, nonsingular information and suitable smoothness; an ancillary statistic specifies the conditioning surface when the fit alone does not exhaust the relevant sample coordinates. In a regular model the unnormalized leading expression has relative error of order locally; normalization and conditioning matter for refined inference. It is not a universal exact density. Under a smooth one-to-one parameter transformation, the Hessian at the fit transforms without a score term, so its determinant supplies precisely the density Jacobian determinant. This proves the invariance of the construction under one-to-one changes of statistical parameters.
Here put and . The log-likelihood, apart from a data-only constant, is
The two score equations give
The sample sums form a sufficient statistic, and there is no need to retain an additional ancillary in calculating their joint density. Inverting the fitted coordinates gives and . With parameter order , the fitted observed information is
Also,
Thus the required joint P-star approximation is
To integrate out , substitute in
Taking and rearranging the powers yields
The beta function identity therefore gives
Both normalized approximations are actually exact. To check the joint assertion independently, and have independent gamma distributions with shape and rates and . Their joint density is . The inverse-coordinate Jacobian determinant is
which reproduces the normalized joint expression. This is the P-star density for two exponential rates; its marginal gives the stated F-distribution. Stirling's approximation gives , so using only the leading normalizer retains the exact shape but not its exact total mass.
The original PDF's displayed density constant is missing a square on . Its printed expression integrates to , rather than one in general. The normalized density above is the one consistent with the stated F-distribution.
For an interest parameter and nuisance parameter , the profile likelihood is , where maximizes the likelihood at fixed . Its maximum occurs at the joint maximum-likelihood estimate of . It preserves the optimized likelihood ratios for different interest values and is unchanged by one-to-one reparametrization of the nuisance at each fixed . It generally is neither a marginal sampling density nor a conditional likelihood, and eliminating nuisance parameters by maximization can introduce appreciable small-sample distortion. Differentiating the constrained score equation gives the curvature formula
at a joint stationary point. The corresponding expected-information expression is the efficient information for the interest parameter. Under the regular conditions of Wilks theorem, twice the profile log-likelihood drop has an asymptotic chi-squared distribution with the dimension of the interest parameter as its degrees of freedom.
For scalar nuisance, the Barndorff-Nielsen modified profile likelihood may be written
where the sample-coordinate derivative holds and the ancillary statistic fixed. For vector nuisance the last two quantities are determinants. An equivalent form is , since differentiating the constrained nuisance score gives . The semicolon denotes differentiation in sample coordinates, not another ordinary parameter derivative. This Jacobian determinant adjustment is essential to the definition.
In the present model set
The log-likelihood is . Consequently
The profile score is , and its derivative is . The expression in parentheses is the variance of the fixed design values under positive weights proportional to . Thus the profile log-likelihood is strictly concave for a nonconstant design. Its score changes from to as goes from to , giving a unique finite fit. If all coincide, only the mean is identifiable and inference on alone is not defined; that exceptional design must be excluded.
The specified ancillary coordinates give . At fixed , therefore,
Meanwhile the constrained observed information is
Substitution proves
Equivalently the sample derivative of the nuisance score is , which equals at the constrained fit and gives the same cancellation. Thus the modified profile likelihood for exponential regression is proportional to the ordinary profile likelihood. This conclusion uses the given ancillary; no proof of its ancillarity is required.
Use for the intercept, to avoid confusing it with the observation means. Let be the fitted Poisson regression mean. Apart from constants, its log-likelihood is
At a finite interior maximum-likelihood estimate, the intercept score equation gives the Poisson regression margin-matching score equations:
The statistical saturated model fits each mean to , taking the boundary mean zero if . Subtracting fitted from saturated log-likelihood gives the Poisson deviance
The intercept balance cancels the sum of linear terms. Consequently
The unsimplified formula makes nonnegativity transparent; individual terms in the simplified formula need not each be positive.
For the deviance goodness-of-fit test, if the fitted design has rank , compare a suitably calibrated with , rejecting for large values. This approximation applies in a regular large-count regime, such as a fixed set of cells with increasing exposure and positive fitted proportions. It is not automatically accurate for many sparse Poisson observations. Check residual patterns and overdispersion; where the approximation is doubtful, simulate independent Poisson random variables with fitted means, refit each sample, and use a parametric bootstrap distribution of the same statistic. A large p-value means no detected lack of fit, not proof of the model. If a finite interior estimate fails to exist, the score proof needs a boundary-limit interpretation; for example all-zero data give a supremum with every fitted mean tending to zero and limiting deviance zero.
Write for the scalar intercept denoted in the question, to distinguish it from the individual means . The log-likelihood of the independent Poisson observations is
Differentiating gives the score equations for an interior maximum-likelihood estimator:
In particular, when ,
The left side is a weighted covariate mean whose derivative is its weighted variance, so it is strictly increasing when the covariates are not all equal. An interior solution is unique if the observed count-weighted mean is strictly between the smallest and largest covariates. At an extreme it may require a limiting slope, and if all counts are zero the maximum occurs as , rather than at finite parameters. Constant covariates make the slope and intercept unidentifiable.
Put . The negative Hessian matrix is deterministic at given parameters and is also the Fisher information matrix:
Under the usual interior, increasing-information regularity for asymptotic normality of a maximum likelihood estimator, replace the unknown means by their fitted values and read the lower-right entry:
Equivalently, Poisson slope information after eliminating an intercept is , with . This explains both the positive denominator for varying covariates and the loss of slope information when their spread is small.
read.table(...,header=T) reads four grouped rows with column names. The displayed outcome counts give the observed risks in the indicated factor order. rbind in the first test pools over predisposition: the two rows represent no cannabis use and some use, with event/non-event counts and . Thus the pooled risks are and . This test concerns the marginal statistical association between use and outcome, not an association adjusted for predisposition.
Under independence, expected counts in the two-by-two contingency table are . The ordinary Pearson chi-squared statistic is . The printed is reproduced by the default two-by-two Yates continuity correction:
All absolute differences here exceed one half. Comparison with a chi-squared distribution on one statistical degree of freedom gives . There is strong evidence of marginal association. The simplified output omits the usual continuity-correction wording, but its numerical value shows which statistic was used.
attach makes the data columns available by name, and tot=with+without obtains group sizes . The grouped binomial regression fits , with response and prior weights . These weights tell R the binomial denominator; treating the four proportions as four equally precise individual observations would be wrong. The default logit link is
Let indicate some use and indicate predisposition. The additive logistic regression specifies . Its log-likelihood, including the binomial constants, is
Differentiation gives score equations and Fisher information matrix , where . Fisher scoring solves these equations, and the inverse fitted Fisher information matrix gives the reported approximate coefficient variances. The number of scoring iterations records numerical convergence, not degrees of freedom or statistical evidence.
The intercept corresponds to no use and no predisposition and gives baseline fitted risk . The additive coefficients imply common adjusted odds ratios:
They multiply the odds, not the risks. The Wald tests divide the estimates by their standard errors and use a standard normal distribution; both predictors have small two-sided p-values. Approximate confidence intervals for these odds ratios are and , obtained by exponentiating the log-scale intervals. The intercept test against zero asks whether the reference risk equals one half, and is not a test of either exposure association. The binomial dispersion is assumed to be one because ; the output does not demonstrate independence or rule out extra-binomial variation.
There are four grouped probabilities and three fitted coefficients, leaving one residual degree of freedom. The fitted risks are approximately . The binomial deviance compares this fit with the four unrestricted observed proportions:
The deviance goodness-of-fit test against has p-value . The common-odds-ratio fit is not rejected at , although the doubly exposed group's observed risk exceeds its fitted risk appreciably and suggests possible interaction.
The second formula adds . Its four coefficients fit all four probabilities, making a saturated statistical model with zero grouped residual degrees of freedom. The near-zero deviance is roundoff from exact interpolation; it supplies no remaining grouped lack-of-fit test. Unlike Q1's unreplicated Gaussian fit, however, these grouped responses represent many independent Bernoulli trials, and their binomial variances are specified by their sizes and probabilities. Coefficient standard errors remain available from binomial information without estimating dispersion from the zero residual degrees of freedom.
Indeed, let be the fitted cell logit link value. The saturated coefficients are
These calculations yield the printed estimates. The delta method gives , and independent groups make the variance of the sum of all eight reciprocal counts. Its square root is , reproducing the printed standard error.
The saturated logistic regression describes the conditional odds ratios as follows:
Thus the main coefficients now concern effects at the other factor's reference level. Logistic interaction as a ratio of odds ratios gives , with approximate confidence interval . The interaction Wald test has , while the nested likelihood-ratio test uses deviance difference on one statistical degree of freedom and has . The data suggest a larger odds association in predisposed subjects, but do not establish this interaction at the conventional level. This conclusion is specific to the odds scale; absolute-risk differences can vary even with no logit-scale interaction.
The Akaike information criterion is . The additive fit has and AIC ; the interaction fit has and AIC . The extra coefficient improves twice the log-likelihood by but incurs penalty , so AIC falls by only . It weakly favors the larger fit rather than decisively establishing an interaction, and AIC is not a p-value.
The marginal odds ratio and the fitted conditional odds ratios are different quantities. Predisposition is associated with both use frequency and outcome, so confounding is a substantive possibility; noncollapsibility of the odds ratio also means marginal and conditional odds ratios need not agree. This is an observational association analysis with only one adjustment factor. It does not by itself establish a causal effect, exclude other confounding, or validate the assumed independent binomial sampling.
For a regular iid model with a fixed-dimensional interior parameter , identifiability, sufficient differentiability and moment domination, and nonsingular per-observation Fisher information , a consistent maximum-likelihood estimator solves the score equation. Expanding it gives
where . The score has mean zero and covariance under the regularity assumptions. This proves the usual large-sample information bound and asymptotic efficiency within regular parametric inference. The observed curvature and the empirical score covariance consistently estimate when the model is correctly specified.
For smooth constraints with full-row-rank derivative , the Wald test uses
The likelihood-ratio test and score test, evaluated using the constrained fit and nuisance-adjusted efficient information, have the same first-order chi-square law. A quadratic local likelihood expansion explains their asymptotic equivalence. Profile likelihood supplies inference about a parameter of interest with fitted nuisance parameters, and the corresponding chi-square cutoffs give approximate confidence regions.
These conclusions depend on the stated regularity: parameters at a boundary, nonidentifiability, singular information, changing dimension, or parameter-dependent support can give different rates and limits. Under misspecification, the fitted parameter generally approaches a pseudo-true value and its variance is a sandwich covariance matrix, not simply inverse Fisher information.
Work at a regular admissible parameter value with , so the score identities in the question apply. For one observation from the exponential dispersion family, the score function and second derivative are
The zero expected score immediately gives . The equality between expected negative curvature and score variance then gives
Thus
The admissible density parameters necessarily make this variance nonnegative; the algebra does not require an unstated sign assumption beyond the regular family and .
With independent observations and canonical parameters , the chain rule gives the vector score function
A finite interior maximum-likelihood estimator satisfies its score equation, proving
Differentiating again, using , gives
This Hessian is independent of the observations, so taking expectation gives exactly the requested Fisher information matrix. In particular, is not the inverse observation-variance matrix: the linear predictor here is the canonical parameter, not the mean.
For the two-group Poisson regression, , , and . Put , , using here for the second group's mean. There are observations, and
Hence the total Fisher information matrix and its inverse are
The theorem on asymptotic normality of a maximum likelihood estimator says that in a regular identifiable model with nonsingular information the estimator is approximately multivariate normal, centred at the true parameter and with covariance equal to the inverse total information. Here one can regard one observation from each group as an independent pair and apply its usual iid version to such pairs. Therefore
For a direct check, the second score equation gives the fitted second-group mean , and the first then gives the fitted first-group mean . Thus the Poisson log-rate ratio estimate is . The delta method, using independent sample-mean variances and , gives the same variance . If a group total is zero, its finite log-mean estimator fails to exist; with fixed positive true means this event has probability tending to zero exponentially as grows.
The logit link gives . Independence in the Bernoulli logistic-regression model therefore gives the log-likelihood
Differentiate using :
The maximum-likelihood estimator, if finite, solves the single score equation . If at least one is nonzero, the log likelihood is strictly concave at finite , so any root is the unique maximizer.
Use Newton method, equivalently one-parameter iteratively reweighted fitting:
A damped step or a bracketed score solver is useful if a full step overshoots. The finite maximum-likelihood estimate in a one-parameter logistic model requires the score limit at minus infinity to be positive and its limit at plus infinity to be negative. Perfect sign separation of the observations can instead send the estimate to an infinite coefficient; if every , the coefficient is unidentifiable. These possibilities should be checked before claiming convergence to a finite estimate.
Independence gives the log-likelihood
Differentiating yields the score equations
When a finite maximum-likelihood estimator exists, multiplying these equations by gives
The qualification about a finite estimator matters under complete separation in logistic regression, where coefficient estimates can escape to infinity.
Insert the Breslow estimator into the fitted cumulative intensities and interchange the finite sums:
Every observed event contributes exactly one after summing its fitted contribution over its risk set. Therefore
This identity is algebraic and holds for any finite coefficient used consistently in the relative hazards and the Breslow estimator; it does not require the score equation to be solved exactly. It can fail if the baseline hazard is estimated or smoothed by a different procedure.
Let
and set
The within-galaxy contrasts contain no , so the parameter-dependent log-likelihood reduces to
The score equations give the maximum-likelihood estimators
The calibrator and Hubble-flow samples are independent, so both estimators are unbiased and
The Fisher information matrix for is
Hence : attains the multiparameter Cramér-Rao bound.
For each substudy define the log-odds treatment effect
The normal random-effects log-likelihood, up to an additive constant, is
Its score equations give
These are the maximum-likelihood estimators rather than the unbiased sample-variance estimator. At an interior solution with , the Hessian in is negative definite, which is the required second-order condition.
Ignoring factors that do not depend on , each observed side-effect contributes and each censored observation contributes . Thus
The score equation gives
This event-count divided by person-time estimator is the sample analogue of the expectation ratio in part iv.