Past exam of the mathematics course of the University of Cambridge / 2022 / iii / Paper 210 (Topics in Statistical Theory)

For $\lambda \in \mathbb{R}$, convexity of $u\mapsto e^{\lambda u}$ on $[0,1]$ givesConsequentlyFor $\lambda \ge 0$, independence and concavity of $u\mapsto \log \{1+u(e^{\lambda }-1)\}$ yieldThe Chernoff bound therefore gives, for $a=\bar{\mu }+x$,For $\bar{\mu }<a<1$, the minimizer satisfiesSubstitution gives the binary relative entropyThe endpoint cases follow by continuity.

Put $a=(1+\delta )\bar{\mu }$. The same calculation, followed by $\log (1-z)\le -z$, givesThusThe supplied lower bound $\log (1+\delta )\ge 2\delta /(2+\delta )$ implieswhich proves the second upper-tail estimate. If $(1+\delta )\bar{\mu }>1$, the event is empty and the same bound remains true.

For the lower tail, apply the exponential-moment argument with $\lambda <0$, or equivalently optimize at $a=(1-\delta )\bar{\mu }$. It givesFinally,for $0\le \delta <1$, yielding $e^{-n{\delta }^2\bar{\mu }/2}$; the case $\delta =1$ follows by a limit.

A kernel for density estimation is an integrable function $K:\mathbb{R}\to \mathbb{R}$ with $\int K=1$. For bandwidth $h>0$, the kernel density estimator isA kernel of order ell has

Let ${\mathcal{H}}_{n,\delta }$ and $\Gamma$ be as in the question, put $h_{\max }=\max {\mathcal{H}}_{n,\delta }$, and define the positive pilot boundFor every $h\in {\mathcal{H}}_{n,\delta }$, formThe Lepski bandwidth selection method takesSmall bandwidths have little bias but wide intervals; the rule increases the bandwidth until the estimates cease to be mutually compatible.

The grid is geometric with ratio two. Its lower endpoint has order $\log (\log n/\delta )/n$, while its upper endpoint is of orderFor fixed $\beta \le \ell$, the oracle bandwidtheventually lies between these endpoints. Its largest grid predecessor ${\tilde{h}}_{\mathrm{opt}}$ therefore exists, and the dyadic spacing gives

On ${\Omega }_0$, the true value $f(x)$ belongs to every interval with $h\le {\tilde{h}}_{\mathrm{opt}}$. Their intersection is therefore nonempty, soThe intervals at ${\hat{h}}_{\delta }$ and ${\tilde{h}}_{\mathrm{opt}}$ have a common point. Since ${\hat{\sigma }}_{n,h,\delta }$ decreases with $h$,Using ${\hat{f}}_{\infty }\le A$ and ${\tilde{h}}_{\mathrm{opt}}\ge h_{\mathrm{opt}}/2$ givesBecause $|{\mathcal{H}}_{n,\delta }|=O(\log n)$, enlarging the constant gives the required

For the final claim take ${\delta }_n=n^{-3}$ for sufficiently large $n$ and use $\hat{h}={\hat{h}}_{{\delta }_n}$. The squared error on ${\Omega }_0$ is at most a constant timesOff ${\Omega }_0$, boundedness of $K$ and the lower endpoint of the bandwidth grid give a deterministic polynomial bound on $|{\hat{f}}_{n,\hat{h},K}(x)|$; multiplying its square by $\mathbb{P}({\Omega }_0^c)\le n^{-3}$ is $O(n^{-1})$. Since $f(x)\le A$ and $N$ is independent of ${\delta }_n$, this term is no larger than the target rate after changing the constant. Enlarging it once more covers the finitely many $n<N$, proving the claimed adaptive mean squared error bound.

PutThe degree-$p$ local polynomial estimator is ${\hat{m}}_n(x)={\hat{b}}_0$, whereDefine the local polynomial Gram matrixAssume that it is positive definite, and writeThe normal equations then giveThus the estimator is a linear estimator in nonparametric regression, and the displayed $w_{p,i}$ are its effective kernel weights.

If $R$ is a polynomial of degree at most $p$, then $R(x+hu)=c^T{r}_p(u)$ for some $c$. Feeding $Y_i=R(x_i)$ into the weighted least-squares problem gives the exact zero-residual fit $c$, whose intercept is $R(x)$. Hence the polynomial reproduction property of local polynomial regression is

Let ${\beta }_0=\lceil \beta \rceil -1$. The Hölder class $\mathcal{H}(\beta ,L)$ consists of functions with derivatives through order ${\beta }_0$ bounded by $L$ andThe subclass $\tilde{\mathcal{H}}(\beta ,L)$ additionally makes every derivative of order below ${\beta }_0$ $L$-Lipschitz.

Suppose $p<{\beta }_0$. Taylor's theorem and that additional Lipschitz condition give, for the degree-$p$ Taylor polynomial $T_p$ at $x$,On the support of $K_i$, $|u_i|\le 1$, soFor the regular design $x_i=i/n$, at most $2nh+1\le 4nh$ points satisfy $|x_i-x|\le h$ when $h\ge 1/(2n)$. Polynomial reproduction cancels $T_p$, and thereforeThus the universal exponent in the question is $\gamma =1$, with the displayed choice of $C({\lambda }_0,p,L,K)$.

The pushforward measure of $\mu$ under $g$ isThe Lebesgue decomposition theorem says that for sigma-finite measures $P,Q$ there are unique measures $P_{\mathrm{ac}}$ and $P_{\mathrm{s}}$ such that

For a convex lower-semicontinuous function $f$ with $f(1)=0$, choose any measure $\lambda$ dominating probability measures $P,Q$, put $p=dP/d\lambda$ and $q=dQ/d\lambda$, and define the f-divergenceusing the lower-semicontinuous perspective value when $q=0$. This definition includes the singular part and is independent of $\lambda$.

To prove the data processing inequality for f-divergences, let $\mathcal{G}=\sigma (g)$ and use $\lambda =P+Q$. The densities of $g_{\ast }P$ and $g_{\ast }Q$, pulled back to $\mathcal{G}$, are ${\mathbb{E}}_{\lambda }(p\mid \mathcal{G})$ and ${\mathbb{E}}_{\lambda }(q\mid \mathcal{G})$. Since the perspectiveis jointly convex, conditional Jensen inequality givesIntegration proves

The chi-squared divergence iswhen $P\ll Q$, and is infinite otherwise.

Fix a probability measure $Q$. Let $J$ be uniform on $\{1,\dots ,M\}$ and, conditionally on $J=j$, draw $X$ from $P_j$. Compare this joint law with the reference law under which $J$ is uniform and independent of $X\sim Q$. Forthe target expectation is $M^{-1}{\sum }_j{P}_j(A_j)$, while its reference expectation isbecause the $A_j$ form a partition. Its reference variance is $M^{-1}(1-M^{-1})$. The Cauchy-Schwarz inequality applied to the likelihood ratio givesThe last divergence separates over $J$:Taking the infimum over all probability measures $Q$ proves the required inequality.