Past exam of the mathematics course of the University of Cambridge / 2010 / iii / Paper 32 / 2 / Solution

One sufficient maximum likelihood estimation statistical consistency theorem is the following. Let the observations be independent random variables with a common density $f_{\theta }$ relative to a fixed measure, let $\Theta$ be a compact metric parameter set containing the true value ${\theta }_0$, and suppose that the expected log-likelihoodis finite, continuous, and uniquely maximized at ${\theta }_0$. Assume also the uniform law of large numbersand assume that a measurable global maximum-likelihood estimator exists. Then every such maximum-likelihood estimator converges in probability to the true parameter. A useful sufficient condition for this uniform law of large numbers is almost-sure continuity of $\theta \mapsto \log f_{\theta }(Y_1)$ on the compact parameter set and an integrable envelope ${\sup }_{\theta }|\log f_{\theta }(Y_1)|\le M(Y_1)$, ${\mathbb{E}}_{{\theta }_0}M(Y_1)<\infty$. Under these conditions, identifiability supplies uniqueness through the Kullback-Leibler divergence identitywith equality only at $\theta ={\theta }_0$.

To see the statistical consistency conclusion, fix $\epsilon >0$. Compactness and the unique maximum give a strictly positive separation gapwhenever the set outside the neighbourhood is nonempty. If ${\Delta }_n$ denotes the supremum discrepancy in the uniform law of large numbers, maximization of the empirical log-likelihood yieldsHence the probability of $d({\hat{\theta }}_n,{\theta }_0)\ge \epsilon$ is at most $\mathbb{P}(2{\Delta }_n\ge {\delta }_{\epsilon })$, which tends to zero. This is the argmin consistency under uniform convergence in probability argument applied to the negative log-likelihood. These are sufficient conditions, not necessary ones; parameter-dependent support can require a direct argument, and independent observations need not have a common distribution in other models.