Past exam of the mathematics course of the University of Cambridge / 2012 / iii / Paper 39 (Statistical Theory)

Write $Ph=\mathbb{E}h(X)$ and $P_{n}h=n^{-1}{\sum }_{i}h(X_i)$ for the expectation and empirical measure. A sufficient bracketing of a function class condition is that, for every $\epsilon >0$, finitely many brackets $[l_j,u_j]$ cover $\mathcal{H}$, with measurable integrable endpoints and $P(u_j-l_j)<\epsilon$. The uniform strong law from finite L1 bracketing then givesHere the observations are independent and identically distributed. For an uncountable class, one either assumes a measurable supremum, for example through a pointwise separable function class, or formulates the conclusion as a pathwise bound on a common probability-one event.

For the parameterized class, the Heine-Borel theorem makes $\Theta$ compact. Put $M(x)={\sup }_{\theta \in \Theta }|q(\theta ,x)|$. A countable dense subset of $\Theta$ gives the same supremum because of continuity, so $M$ is measurable. Define the modulusThe supremum is measurable by taking a countable dense subset of the compact set of admissible pairs. For each $x$, uniform continuity on $\Theta$ gives $w_{\delta }(x)\to 0$. Also $w_{\delta }\le 2M$, and $PM<\infty$. Thus the dominated convergence theorem gives $P{w}_{\delta }\to 0$.

Choose a finite net of radius $\delta$ ${\theta }_1,\dots ,{\theta }_N$ in $\Theta$. The lower and upper envelopes of $q(\theta ,\cdot )$ over each closed ball $\Theta \cap \overline{B}({\theta }_j,\delta )$ are measurable integrable function brackets; the same countable-dense-set argument applies within each such compact ball. Their widths are at most $w_{2\delta }$. They cover the whole class, so its bracketing of a function class condition follows. Moreover, dominated convergence shows that $\theta \mapsto Pq(\theta ,\cdot )$ is continuous. Both empirical and population functions therefore have a supremum over a common countable dense parameter set. Applying the uniform strong law from finite L1 bracketing proves uniform almost-sure convergence over the entire compact parameter set, rather than merely convergence at each fixed parameter.

For the exponential family,where $A={\sup }_{\Theta }|\theta |$ and $B={\sup }_{\Theta }|K(\theta )|$. Thus a weak sufficient condition is bounded $K$ on $\Theta$, $f_0(X)>0$ almost surely, and integrability of $|X|+|\log f_0(X)|$ under the sampling law. No positive lower bound on $f_0$ over the whole real line is required. Zeros off the sampling support are harmless: choose arbitrary finite versions of the log-densities on that common null set when applying the function-class theorem.

If the sampling law is $f({\theta }_0,\cdot )$, one convenient assumption is that $\Theta$ is a compact subset of the interior of the finite domain of the cumulant function of an exponential family, andOn that interior, $K$ is continuous, hence bounded on $\Theta$. Finiteness of $K$ at ${\theta }_0\pm a$ for some $a>0$ implies ${\mathbb{E}}_{{\theta }_0}{e}^{a|X|}<\infty$, hence ${\mathbb{E}}_{{\theta }_0}|X|<\infty$. The displayed integral gives the remaining integrability. If the observations come from an arbitrary unrelated law, assumptions on $K$ and $f_0$ alone cannot control that law's tails; the sampling-law integrability must then be stated explicitly.

At an interior statistical parameter $\theta$, local asymptotic normality means that there are random vectors ${\Delta }_{n,\theta }$ and a symmetric nonnegative Fisher information matrix $I_{\theta }$ such that, for each fixed $h\in {\mathbb{R}}^p$,A common stronger definition requires this expansion also for $h_n\to h$. The fixed-$h$ version suffices here. A regular identifiable model usually has positive definite Fisher information; nonsingularity is unnecessary for the contiguity conclusion.

Here are sufficient differentiability conditions for a proof by Taylor's theorem. Suppose the local probability density functions have common support, ${\ell }_{\vartheta }(y)=\log f(\vartheta ,y)$ is twice continuously differentiable near $\theta$, differentiation can pass through the normalization integral twice, the score function $s_{\theta }=\nabla {\ell }_{\theta }$ has finite second moment, and the local Hessian matrix is bounded in operator norm by an integrable envelope of a function class. These hypotheses implyThe first identity differentiates $\int f_{\theta }=1$ once; the second uses ${\nabla }^2{f}_{\theta }=f_{\theta }({\nabla }^2{\ell }_{\theta }+s_{\theta }{s}_{\theta }^T)$ and differentiates twice. The central limit theorem gives ${\Delta }_{n,\theta }=n^{-1/2}{\sum }_i{s}_{\theta }(Y_i)\stackrel{d}{\to }{N}_p(0,I_{\theta })$. Taylor's integral remainder givesThe weak law of large numbers and the continuity and integrable-envelope hypotheses make the bracket converge in probability to $-I_{\theta }/2$. Indeed, the expectation of the supremum of the Hessian matrix difference over a shrinking neighborhood tends to zero by dominated convergence, and Markov inequality controls its empirical average. This proves local asymptotic normality and explains the role of both derivatives.

Write $Q_n◃P_n$ for contiguity of probability measures: for every measurable sequence $A_n$, $P_n(A_n)\to 0$ implies $Q_n(A_n)\to 0$. Mutual contiguity requires this in both directions. A useful form of Le Cam's first lemma is the following: if the Radon-Nikodym derivative $L_n$ of the absolutely continuous part of $Q_n$ relative to $P_n$ converges in distribution under $P_n$ to $L$, with $L>0$ almost surely and $\mathbb{E}L=1$, then the two sequences are mutually contiguous. This includes the usual case $L_n=d{Q}_n/d{P}_n$. The mean-one condition ensures uniform integrability of $L_n$ and vanishing mass of any singular part; positivity ensures the reverse implication.

Take $P_n=P_{\theta }^n$ and $Q_n=P_{\theta +h/\sqrt{n}}^n$. Their likelihood ratio on the support of $P_n$ satisfies, by local asymptotic normality and the continuous mapping theorem,This limit is positive and $\mathbb{E}L=e^{-v/2}{e}^{v/2}=1$, including $v=0$. The conditions of Le Cam's first lemma therefore hold, giving mutual contiguity of the product sampling laws. The products, rather than the one-observation laws, are the relevant sequences.

For any $\epsilon >0$, let $A_n=\{\Vert {\hat{\theta }}_n-\theta \Vert >\epsilon \}$. Consistency under $P_{\theta }^n$ makes $P_{\theta }^n(A_n)\to 0$, and contiguity gives $Q_n(A_n)\to 0$. Thus the estimator remains consistent for $\theta$ under each fixed local alternative. Since $\Vert \theta +h/\sqrt{n}-\theta \Vert \to 0$, it is also consistent for the moving parameter in the sense that $\Vert {\hat{\theta }}_n-(\theta +h/\sqrt{n})\Vert \to 0$ in probability under $Q_n$.

Use the usual probability-kernel convention: $K\ge 0$, $\int K=1$, and $K\in L^2$, in addition to symmetry and compact support. These conditions are needed for the stated finite bound; symmetry and compact support by themselves do not impose normalization or nonnegativity. Interpret the differentiability assumption in the usual Sobolev space sense $f\in H^2(\mathbb{R})$, or assume classical regularity sufficient for the integral Taylor remainder. The kernel density estimator isThe bias-variance decomposition of mean squared error, followed by Tonelli theorem, yieldsIndependence and a change of variables give the exact integrated variance of a kernel density estimatorAll changes of integration order are justified either by nonnegativity or by the displayed finite integrals.

For the bias of a kernel density estimator, symmetry gives $\int uK(u)du=0$. Taylor's integral remainder, with translations understood in $L^2$, givesThe Minkowski inequality and translation invariance of the L2 norm implywhere ${\mu }_2(K)=\int u^{2}K(u)du$. Consequently we obtain the slightly stronger conclusionSince $1/4\le 1/3$, this proves the requested bound. For signed kernels, the same argument has $\int u^2|K(u)|du$ in place of ${\mu }_2(K)$; the printed bound is not generally justified merely by signed-kernel symmetry. For example, let $U_a=(2a{)}^{-1}{1}_{[-a,a]}$ and $K=(4/3)U_1-(1/3)U_2$. This symmetric compactly supported signed kernel has integral one and ${\mu }_2(K)=0$, but its fourth moment is nonzero. For a normal density its convolution has nonzero bias at any fixed positive bandwidth. Sending $n\to \infty$ would therefore contradict a bound containing only the integrated variance term.

Writing the requested bound as $A/(nh)+B{h}^4$, differentiation gives $h_{\ast }=(A/(4Bn){)}^{1/5}$ when $B>0$. Thus $h\asymp n^{-1/5}$ gives mean integrated squared error of order $n^{-4/5}$. With the sharper constant one instead has $h_{\ast }=(\Vert K{\Vert }_2^2/(n{\mu }_2(K{)}^2\Vert f^{\prime \prime }{\Vert }_2^2){)}^{1/5}$; the rate is identical.

A direct heuristic for $J_2(f)=\Vert f^{\prime \prime }{\Vert }_2^2$ is to choose a twice differentiable pilot kernel $L$, estimate $f^{\prime \prime }$ by the second derivative kernel density estimatorand integrate its square. Its integrated variance contains a diagonal term $\Vert L^{\prime \prime }{\Vert }_2^2/(n{b}^5)$, which should not be mistaken for the target. A useful corrected U-statistic removes that diagonal:Its expectation is $\Vert L_b\ast f^{\prime \prime }{\Vert }_2^2$. Thus smoothing bias of an estimator, rather than the diagonal contribution, remains. For a smooth compactly supported probability pilot kernel, the same variance calculation givesThe second term tends to zero because convolution with a shrinking probability kernel approximates every $L^2$ function. Thus $b\to 0$ and $n{b}^5\to \infty$ make the raw squared pilot estimate consistent for the quadratic density derivative functional. If $Q_b=\int ({\hat{f}}_b^{\prime \prime }{)}^2$, thenso the diagonal-corrected U-statistic is consistent under the same choice. Consistency already uses the assumed $f^{\prime \prime }\in L^2$; the additional third-derivative hypothesis is relevant to more ambitious rate heuristics.

Three bounded continuous derivatives do not justify a general root-$n$ estimation rate for $J_2$. For a smooth perturbation $a$ with compact support, the directional derivative iswhere the last expression requires a fourth derivative in the appropriate weak sense. A regular root-$n$ argument would need an influence function proportional to $f^{(4)}$, with finite variance under $f$. Three derivatives do not supply that condition.

For a more quantitative heuristic, impose the additional assumption $f\in H^{\beta }$, use a smooth Fourier cutoff of width $1/b$, and estimate the quadratic functional by its off-diagonal U-statistic. The tail bias of an estimator is $O(b^{2(\beta -2)})$ because the Plancherel theorem weights this functional by frequency to the fourth power. The degenerate part of its standard deviation is $O(n^{-1}{b}^{-9/2})$: the squared $L^2$ norm of the quadratic kernel scales as $b^{-9}$. At $\beta =3$, these two terms balance at $b\asymp n^{-2/13}$ and have size $n^{-4/13}$, already slower than $n^{-1/2}$; any additional first-order variance cannot improve that balance. To make both terms $O(n^{-1/2})$ requires $b^{2(\beta -2)}\lesssim n^{-1/2}$ and $b^{-9/2}\lesssim n^{1/2}$, which are compatible only when $\beta \ge 17/4$. This is an illustrative smoothness calculation under extra Sobolev and moment assumptions, not a rate theorem asserted solely from bounded third derivatives. Particular much smoother probability density functions may of course permit root-$n$ estimation.

There is a genuine qualification: the first printed inequality is valid as a useful general assertion for $0<\eta <1$, but is false for arbitrary $\eta >0$. Also the displayed likelihood ratio presupposes $P_{f_1}\ll P_{f_0}$. We first prove the intended bound and then give a counterexample to the unrestricted quantifier.

For any estimator $T$, set $A=\{d(T,f_0)<r_n\}$. By the triangle inequality, $d(T,f_1)\ge r_n$ on $A$, while $d(T,f_0)\ge r_n$ on $A^c$. HenceLet $C=\{|Z-1|\le \eta \}$ with $0<\eta <1$. On $C$, $Z\ge 1-\eta$. ThereforeThe last step is Markov inequality. Taking the supremum over the model and then the infimum over estimators proves the intended metric two-point risk bound. At $\eta =1$ the right side is zero. For $\eta >1$ a negative right side is also harmless, but a positive right side need not be a lower bound.

Here is an explicit counterexample with positive probability density functions and $n=3$. Let $A_0=[0,1/2)$, $\delta =1/100$, and take the model consisting ofUse the L2 norm metric, for which $D=d(f_0,f_1)=2(1-2\delta )$, and take $r_n=D/2$ for every $n$. The majority-half decision selects $f_0$ if at least two observations lie in $A_0$. Each error probability is $p=3{\delta }^2-2{\delta }^3=0.000298$, so its maximum normalized risk function is $2p=0.000596$. This is also the likelihood-ratio test, giving ${\mathbb{E}}_{f_0}|Z-1|=2(1-2p)$. At $\eta =3/2$ the proposed right side is $1/12-2p/3>0.083$, contradicting the risk of this explicit estimator. Thus the intended range restriction cannot be omitted.

For the second request use the intended L2 norm metric $d(f,g)=({\int }_0^1(f-g{)}^2{)}^{1/2}$. The PDF defines its square, not the square itself as a metric. PutBoth probability density functions belong to the model, and $d(f_0,f_{1,n})=a_n=2r_n$. The two-point choice may depend on $n$, because the lower bound is applied separately at each sample size. Under $f_0$, the product likelihood ratio $Z_n={\prod }_i(1+a_{n}g(X_i))$ satisfiesThis is the product chi-squared divergence. By Cauchy-Schwarz inequality, ${\mathbb{E}}_0|Z_n-1|<1/4$. With $\eta =1/2$, the proved metric two-point risk bound has normalized right side greater than $1/8$. ConsequentlyThe proof works for every $n\ge 1$ despite the introductory $n>2$. Measurability of the functions defining the probability density functions is understood. This establishes the requested lower bound; it does not assert a matching upper bound over this unrestricted class.

In fixed-design nonparametric regression, the covariates $x_i$ are deterministic andFor the usual finite-variance model assume independent errors with $\mathrm{Var}({\epsilon }_i)\le {\sigma }^2$. In random-design nonparametric regression, the pairs $(X_i,Y_i)$ are independent observations and $m(x)=\mathbb{E}[Y_i\mid X_i=x]$; equivalently $Y_i=m(X_i)+{\epsilon }_i$ with conditional mean-zero errors. Conditional variances are generally assumed bounded for risk bounds.

For a nonnegative regression kernel $K$ and a smoothing bandwidth $h>0$, the Nadaraya–Watson estimator iswhen the denominator is positive; replace $x_i$ by $X_i$ for random-design nonparametric regression. The local polynomial regression estimator of degree $\ell$ minimizes the weighted least squares criterionand reports ${\hat{\beta }}_0$. If the weighted Gram matrix is invertible, this minimizer is unique. The Nadaraya-Watson estimator is exactly local polynomial regression of degree zero. Higher degrees reproduce nonconstant polynomials and reduce boundary bias of an estimator.

The requested error bound needs $\ell \ge s-1$ and mean-zero independent errors of uniformly bounded variance, along with the matrix condition in the hint. The degree requirement is missing from the printed question. To see the problem, take $\ell =0$, $K=1_{[-1,1]}$, $m(t)=t$, zero errors, $x=0$, and $h=n^{-1/4}$. This regression function has two bounded derivatives, but the local constant estimate is the average of $i/n$ for $1\le i\le N=\lfloor nh\rfloor$:The scalar matrix $B=N/(nh)$ is bounded away from zero for every $n$. Nevertheless $(nh{)}^{-1/2}+h^2=n^{-3/8}+n^{-1/2}=o(h)$, so the printed conclusion with $s=2$ fails. A bounded-variance error hypothesis is also essential; otherwise an expectation bound need not even be finite.

Under the qualified hypotheses, use $U(t)=(1,t,\dots ,t^{\ell }/\ell !{)}^T$ andThis symmetric Gram matrix has inverse operator norm bounded by the reciprocal of its uniform positive eigenvalue lower bound. If $K$ is supported in $[-R,R]$, only $|x_i-x|\le Rh$ contribute. The equally spaced design and $nh\ge 1$ imply that there are at most $2Rnh+1\le Cnh$ such points. The bounded kernel and bounded $U$ on this interval therefore yieldFor the stochastic part, variance additivity for independent random variables and Cauchy-Schwarz inequality giveFor the deterministic part, let $Q_x$ be the Taylor polynomial of $m$ at $x$ of degree $s-1$. The polynomial reproduction property of local polynomial regression gives ${\sum }_i{W}_{ni}(x)Q_x(x_i)=Q_x(x)=m(x)$. By Taylor's theorem, $|m(x_i)-Q_x(x_i)|\le C|x_i-x{|}^s\le C{h}^s$ on the kernel window. ThusCombining the two parts provesThe argument also applies at the endpoints when the assumed matrix bound holds. With arbitrary degree, the generally valid smoothness term is $h^{\min (s,\ell +1)}$; extra symmetry can improve some interior biases but does not fix the general boundary counterexample.

A map $\Phi :S_D\to \mathbb{R}$ is Hadamard differentiable at $s_0$ if there is a continuous linear functional ${\dot{\Phi }}_{s_0}:S\to \mathbb{R}$ such that, whenever $t_j\to 0$ with $t_j\ne 0$, $a_j\to a$ in $S$, and $s_0+t_j{a}_j\in S_D$,For tangential Hadamard differentiability, the limit directions $a$ are restricted to a specified tangent subspace and the derivative is continuous linear on that subspace. The domain condition is indispensable.

Assume $X_n\in S_D$ almost surely and use the usual measurable, separably supported setting for norm-valued convergence in distribution, with the limit in the tangent space. The functional delta method givesTo justify this by Skorokhod representation theorem, a sufficient precise version is: for weakly convergent random elements in a separable complete metric space, one can construct copies of the whole sequence and its limit on one probability space which converge almost surely in the metric. More generally the separable-limit-support version of the representation theorem can be used. Apply it to $Z_n=r_n(X_n-s_0)$, obtaining ${\tilde{Z}}_n\to \tilde{X}$ almost surely. The variables $s_0+r_n^{-1}{\tilde{Z}}_n$ have the same laws as $X_n$ and lie in $S_D$ on a common probability-one event. The definition of Hadamard differentiability, with $t_n=1/r_n$, gives convergence almost surely of the transformed quotients to ${\dot{\Phi }}_{s_0}(\tilde{X})$, and hence the claimed convergence in distribution. If $X$ is a Gaussian measure, the limit is Gaussian because the derivative is linear. Without membership of $X_n$ in $S_D$, the expression $\Phi (X_n)$ in the question is not defined.

The final function space is $C_b(\mathbb{R})$ with supremum norm; it is not separable as a whole. This does not affect the deterministic differentiability proof below. Any statistical application of the preceding representation argument must use its separability or tightness hypotheses, rather than assuming that every normed space automatically satisfies them.

Fix admissible $F,G$ and set $T(F,G)={\int }_0^{u}G(x)F^{\prime }(x)dx$. The candidate derivative, defined even for continuous directions $a,b$ without derivatives, isThis follows formally from the product rule and integration by parts. We now establish the uniform-direction limit required by Hadamard differentiability.

Let $F_j=F+t_j{a}_j$ and $G_j=G+t_j{b}_j$ belong to the stated domain, with $a_j\to a$, $b_j\to b$ in supremum norm. The domain gives $\int |F_j^{\prime }|\le 1$ and $\int |F^{\prime }|\le 1$, while $F_j\to F$ uniformly. We first claim that, for every continuous $b$ on $[0,u]$,For a continuously differentiable test function $v$, integration by parts gives convergence directly from uniform convergence of $F_j$: both boundary terms and $\int v^{\prime }{F}_j$ converge. By the Weierstrass approximation theorem, approximate $b$ uniformly on $[0,u]$ by a polynomial $v$. The difference between the two integrals introduced by this approximation is at most $2\Vert b-v{\Vert }_{\infty }$, by the uniform bounded variation bounds. Letting the approximation error decrease proves the claim. This is weak convergence of bounded-variation integrators.

Now decompose the quotient in the order that retains the bounded-variation bound:The first term differs from $\int b{F}_j^{\prime }$ by at most $\Vert b_j-b{\Vert }_{\infty }$, so it tends to $\int b{F}^{\prime }$. For the second term, integration by parts yieldswhich tends to the corresponding expression with $a$. Hence the quotient tends to ${\dot{T}}_{F,G}(a,b)$ for every admissible perturbation sequence. Finally,This proves that the derivative is a continuous linear functional on the product space. Therefore $T$ is Hadamard differentiable at every point of the stated domain. The bounded-variation restriction is what controls the apparently dangerous interaction between uniformly small perturbations and possibly large perturbation derivatives; no convergence of $a_j^{\prime }$ or $b_j^{\prime }$ has been assumed.