Past exam of the mathematics course of the University of Cambridge / 2006 / ii / Paper 4 / 27J (Principles of Statistics)

The strong law of large numbers states that for independent identically distributed real variables with $\mathbb{E}|X_1|<\infty$, the sample average converges almost surely to $\mathbb{E}{X}_1$. The central limit theorem states that if additionally $0<\mathrm{Var}{X}_1={\sigma }^2<\infty$, thenThe vector form uses finite second moments and gives a centred multivariate Gaussian with the covariance matrix of one observation.

Assume an identifiable smooth parametric model, an interior true parameter, a consistent interior maximum-likelihood estimator, interchange of differentiation and integration, a uniform local law of large numbers for the Hessian, and nonsingular finite Fisher information. For one observation define the score function $s_{\theta }(X)={\nabla }_{\theta }\log p_{\theta }(X)$ andThe score function has expectation zero by differentiating the normalization of the density. Its sample sum satisfies $n^{-1/2}{S}_n(\theta )\stackrel{d}{\to }N(0,J)$ by the vector central limit theorem. The score function equation and Taylor's formula giveThe bracket divided by $n$ converges in probability to $-J$ by consistency and the assumed uniform law. Solving the equation and applying Slutsky theorem yieldsThe integral Hessian avoids incorrectly assuming a single intermediate point for a vector-valued Taylor formula.

Writing $S=\sum X_i$, the likelihood function is ${\theta }^S(1-\theta {)}^{n-S}$ and its supremum occurs at $\hat{\theta }=S/n$, including the limiting boundary cases zero and one. The likelihood ratio compares $2^{-n}$ under the null with that supremum. ThusUse $0\log 0=0$ at boundary outcomes. Wilks theorem gives asymptotic ${\chi }_1^2$ under the regular interior null, because one parameter is constrained. Directly, setting $\hat{\theta }=1/2+Z$ and expanding the bracket gives $2Z^2+O(Z^4)$, henceUnder the null, $2\sqrt{n}{Z}_n\stackrel{d}{\to }N(0,1)$ and $Z_n=O_P(n^{-1/2})$. The remainder vanishes in probability, proving the same chi-squared distribution limit. If an observed $Z_n$ is not small, the quadratic numerical approximation is unreliable and the exact statistic should be used; exact binomial calibration is available. Persistent non-small deviations under a fixed alternative instead make the statistic grow linearly with $n$, rather than having the null chi-squared distribution law.