Past exam of the mathematics course of the University of Cambridge / 2010 / iii / Paper 32 / 2

One sufficient maximum likelihood estimation statistical consistency theorem is the following. Let the observations be independent random variables with a common density $f_{\theta }$ relative to a fixed measure, let $\Theta$ be a compact metric parameter set containing the true value ${\theta }_0$, and suppose that the expected log-likelihoodis finite, continuous, and uniquely maximized at ${\theta }_0$. Assume also the uniform law of large numbersand assume that a measurable global maximum-likelihood estimator exists. Then every such maximum-likelihood estimator converges in probability to the true parameter. A useful sufficient condition for this uniform law of large numbers is almost-sure continuity of $\theta \mapsto \log f_{\theta }(Y_1)$ on the compact parameter set and an integrable envelope ${\sup }_{\theta }|\log f_{\theta }(Y_1)|\le M(Y_1)$, ${\mathbb{E}}_{{\theta }_0}M(Y_1)<\infty$. Under these conditions, identifiability supplies uniqueness through the Kullback-Leibler divergence identitywith equality only at $\theta ={\theta }_0$.

To see the statistical consistency conclusion, fix $\epsilon >0$. Compactness and the unique maximum give a strictly positive separation gapwhenever the set outside the neighbourhood is nonempty. If ${\Delta }_n$ denotes the supremum discrepancy in the uniform law of large numbers, maximization of the empirical log-likelihood yieldsHence the probability of $d({\hat{\theta }}_n,{\theta }_0)\ge \epsilon$ is at most $\mathbb{P}(2{\Delta }_n\ge {\delta }_{\epsilon })$, which tends to zero. This is the argmin consistency under uniform convergence in probability argument applied to the negative log-likelihood. These are sufficient conditions, not necessary ones; parameter-dependent support can require a direct argument, and independent observations need not have a common distribution in other models.

The likelihood function for the shifted exponential distribution isIt increases strictly up to the smallest observation and is zero beyond it. Thus shifted exponential maximum likelihood givesFor $t\ge 0$, independence and the survival function of the exponential distribution give the exact order statistic lawIn particular, ${\hat{\theta }}_n\ge \theta$ almost surely and ${\mathbb{P}}_{\theta }(|{\hat{\theta }}_n-\theta |>\epsilon )=e^{-n\epsilon }\to 0$. This proves statistical consistency. The exact endpoint limit for a shifted exponential distribution is stronger than an asymptotic approximation:The limit is one-sided and nonnormal; the moving support endpoint explains why a regular square-root-$n$ asymptotic normality of a maximum likelihood estimator theorem does not apply here.

Set $S_n={\sum }_{i=1}^n{Y}_i$ and $a_n={\sum }_{i=1}^n{2}^{-i}=1-2^{-n}$. For the Poisson distribution observations, factors independent of $\theta$ can be removed from the likelihood function, leavingWhen $S_n>0$, the log-likelihood derivative is $S_n/\theta -a_n$ and its second derivative is negative. The unique positive maximum-likelihood estimator is thereforeThere is a genuine boundary qualification: if $S_n=0$, the likelihood decreases strictly on $\theta >0$, so no maximum is attained in the stated open parameter space. On the closure $\theta \ge 0$, its maximizer is $0$. Defining the usual extended maximum-likelihood estimator by the same displayed formula for every $S_n$ allows its statistical consistency to be examined; the all-zero event cannot be ignored because its probability does not tend to zero.

Indeed, sums of independent Poisson distribution variables give $S_n\sim \mathrm{Poi}(\theta a_n)$, soThus the extended maximum-likelihood estimator is inconsistent. More precisely, $S_n$ increases to $S_{\infty }={\sum }_{i\ge 1}{Y}_i$, and $\mathbb{E}{S}_{\infty }=\theta <\infty$ shows that this total count is finite almost surely. Its distribution is Poisson distribution with mean $\theta$, by taking limits of the finite-sum laws. HenceThis finite-exposure Poisson inconsistency leaves a nondegenerate random limit even with infinitely many observations. Correspondingly, the total Fisher information is $a_n/\theta \to 1/\theta$, rather than diverging. The common-distribution hypothesis in the introductory maximum likelihood estimation theorem is absent here.