Past exam of the mathematics course of the University of Cambridge / 2014 / iii / Paper 30 (Statistical Theory)

For independent identically distributed observations $Y_1,\dots ,Y_n$, the likelihood function and log-likelihood areA maximum-likelihood estimator is a measurable choice ${\hat{\theta }}_n\in {\mathrm{argmax}}_{\theta \in \Theta }{L}_n(\theta )$, when a maximizer exists. It need not be unique. The one-observation score function is $s_{\theta }(y)={\nabla }_{\theta }\log f(y,\theta )$, and the Fisher information matrix isUnder the regularity assumptions, differentiation beneath the integral gives ${\mathbb{E}}_{\theta }{s}_{\theta }={\nabla }_{\theta }\int f=0$. Differentiating again gives the second information identity. Independent observations have total Fisher information $nI(\theta )$.

Assume fixed parameter dimension, an interior true parameter ${\theta }_0$, a positive-definite matrix $I({\theta }_0)$, the standard differentiability and integrability conditions, and statistical consistency of ${\hat{\theta }}_n$. Then the asymptotic normality of a maximum likelihood estimator isHere $I$ is the information per observation. Thus the leading covariance matrix of the estimator itself is $I({\theta }_0{)}^{-1}/n$.

To prove this, statistical consistency places the estimator in an interior ball about ${\theta }_0$ with probability tending to one, where its score function vanishes. Write $d_n={\hat{\theta }}_n-{\theta }_0$ and use the integral first-order Taylor formula for a vector map:This integral matrix is needed in a vector problem; one does not have to assert a common scalar mean-value point for every score component. PutThe regular local uniform law of large numbers, continuity of the expected Hessian matrix, and statistical consistency give $J_n\stackrel{P}{\to }I({\theta }_0)$. The multivariate central limit theorem givesThe limiting information matrix is nonsingular, so $J_n^{-1}\stackrel{P}{\to }I({\theta }_0{)}^{-1}$. Solving the Taylor identity and applying the Slutsky theorem proves the displayed normal limit, since $I^{-1}I{I}^{-1}=I^{-1}$. Boundary parameters or singular information are outside this regular theorem.

Represent the subset model by a set $M$ of $k$ column indices and let $X_M$ contain those columns. Its ordinary least squares estimator sets the omitted coefficients to zero and hasFor $k=0$, use $P_M=0$. Full column rank of $X$ implies full column rank for each $X_M$. The matrix $P_M$ is an orthogonal projection matrix of rank $k$.

Put $\mu =X\theta$ and $b_M=(I-P_M)\mu$. Projection orthogonality gives the bias-variance decomposition of mean squared errorwhile the expected residual sum of squares isThese equations distinguish the mean-vector prediction risk in this problem from the additional noise in predicting a new response vector.

First take $p<n$. Let $P_X$ project onto the full column space of $X$. Since the full model contains $\mu$, the residual estimate of Gaussian noise varianceis unbiased: its numerator has expectation $(n-p){\sigma }^2$. Consequently an unbiased Gaussian projection risk estimate isIndeed its expectation is $\Vert b_M{\Vert }^2+(n-k){\sigma }^2+(2k-n){\sigma }^2=R(M)$ for every $\theta$, even when the subset model omits true effects. Independence of the two terms is not needed for this expectation calculation. The full-model variance estimate is important: the subset residual variance generally includes omitted-variable bias.

For a finite candidate collection, choose a model minimizing $\hat{R}(M)$, breaking ties in favour of a smaller model. The term $-n{\hat{\sigma }}^2$ is common to all models, so this amounts to minimizingthe Mallows Cp form. The fit improvement competes with an increasing dimension penalty. Unbiasedness holds for each fixed model; the estimate at the data-selected model need not remain unbiased, because selection favours downward fluctuations. The method is a heuristic model selection rule, rather than an assertion that it finds the true model with certainty.

The printed $p\le n$ includes a genuine qualification. If $p=n$, there are no full-model residual degrees of freedom. Except in the balanced case $2k=n$, no integrable data-only unbiased estimator can satisfy the requested identity for all means and unknown variances. Here is a proof of the unknown-variance risk estimation in a saturated Gaussian model obstruction. Since $X$ is invertible, $\mu$ ranges over all ${\mathbb{R}}^n$. Suppose $T(Y)$ had the required expectation at every $\mu$ and variance $s>0$. For $v>0$, let $U\sim N_n(\mu ,vI)$ and let $Y\mid U\sim N_n(U,sI)$. Marginally $Y\sim N_n(\mu ,(s+v)I)$. Applying the assumed identity conditionally giveswhereas applying it to the marginal distribution gives $\Vert b_M{\Vert }^2+k(s+v)$. Their difference is $(n-2k)v$, which must vanish. Integrability under the marginal Gaussian justifies conditioning. If $2k=n$, the obstruction disappears and the residual sum of squares alone has expectation $\Vert b_M{\Vert }^2+k{\sigma }^2$. ThusThe general construction therefore requires $p<n$, or an independently available unbiased variance estimate.

The ordinary column Gram matrix is $G=X^{T}X$. Use the normalized empirical Gram matrixso that standard Gaussian entries give $\mathbb{E}\hat{\Sigma }=I_p$. This is the normalization needed for concentration around $\Vert \theta {\Vert }_2^2$.

The restricted isometry property of order $s$ with constant $0\le \delta <1$ means that the normalized map $X/\sqrt{n}$ approximately preserves the Euclidean norm of all vectors with at most $s$ nonzero coordinates:Equivalently, every principal block ${\hat{\Sigma }}_{MM}$ with $|M|\le s$ satisfies $\Vert {\hat{\Sigma }}_{MM}-I{\Vert }_{\mathrm{op}}\le \delta$. The least such $\delta$ is its restricted isometry constant. In the unnormalized definition apply this property to $X/\sqrt{n}$ itself.

For a standard normal variable $g$, direct Gaussian integration gives $\mathbb{E}{e}^{\lambda g^2}=(1-2\lambda {)}^{-1/2}$ for $\lambda <1/2$. Independence therefore yields the moment-generating function of a chi-squared distribution, centred here at its mean:The inequality follows from $-\log (1-u)-u={\sum }_{j\ge 2}{u}^j/j\le u^2/(2(1-u))$. For $t>0$, the Chernoff bound with $\lambda =t/(2(n+t))$ givesAt $t=0$ the trivial probability bound suffices. This proves the requested bound, with a stronger prefactor one.

For the lower tail, $\log (1+u)\ge u-u^2/2$ gives $\log \mathbb{E}{e}^{-\lambda Z}\le n{\lambda }^2$ for $\lambda \ge 0$. Taking $\lambda =t/(2n)$ yields $\mathrm{Pr}(Z\le -t)\le e^{-t^2/(4n)}$. Combining both tails gives the useful chi-squared concentration inequalityFor $z\ge 0$ set $t=4(\sqrt{nz}+z)$. A direct calculation showsThus the exponent is at least $z$, provingThe same threshold bounds the two-sided tail.

Finally fix a deterministic $\theta$ and put $r=\Vert \theta {\Vert }_2$. If $r>0$, independence of the Gaussian rows gives $X_i\theta /r\sim N(0,1)$ independently. ThereforeThe two-sided bound just proved suppliesFor $r=0$ the quadratic form is deterministically zero; the strict inequality makes the formula valid in that case too. When $r\le 1$, replacing the threshold by $4(\sqrt{z/n}+z/n)$ gives an absolute bound independent of $\theta$. In particular every fixed such direction concentrates at rate $n^{-1/2}$ for a fixed confidence level. The restricted isometry property requires a simultaneous statement over sparse directions; this fixed-direction calculation alone is not that stronger assertion.

Let ${\Delta }_n={\sup }_{\theta \in \Theta }|Q_n(\theta )-Q(\theta )|$. For any $\epsilon >0$, the setis compact. If it is empty there is nothing to prove. Otherwise continuity and the unique minimum give a strictly positive separation gapTake any measurable attained minimizer of $Q_n$. Its defining inequality givesConsequentlyThis proves argmin consistency under uniform convergence in probability. No continuity of $Q_n$ is needed once the minimizer exists; compactness and continuity concern the deterministic separation gap. The uniform convergence in probability assumption controls all candidate parameters, including the random minimizer.

For the estimating equation, fix $\epsilon >0$ and put $a={\theta }_0-\epsilon$, $b={\theta }_0+\epsilon$. The prescribed signs make $\eta =\frac{1}{2}\min \{-S(a),S(b)\}>0$. Pointwise convergence in probability at just these two points impliesand similarly $\mathrm{Pr}(S_n(b)\le 0)\to 0$. With probability tending to one, $S_n(a)<0<S_n(b)$. The intermediate value theorem then gives a zero inside $(a,b)$, and uniqueness identifies it with ${\hat{\theta }}_n$. HenceThis is consistency of a uniquely bracketed zero. It needs no monotonicity, no continuity of the limit $S$, and no uniform convergence of the $S_n$. The bracket interval must lie in the domain of $S_n$: read literally, the printed sign condition for every positive $\epsilon$ puts every real point into $\Theta$, so this requirement is satisfied. More generally it is enough to have an interval about ${\theta }_0$ and sign brackets arbitrarily close to it.