Past exam of the mathematics course of the University of Cambridge / 2021 / iii / Paper 210 (Topics in Statistical Theory)

A centered random variable is sub-Gaussian with parameter ${\sigma }^2$ whenFor $\lambda >0$, the Chernoff bound givesMinimizing at $\lambda =x/{\sigma }^2$ yields $e^{-x^2/(2{\sigma }^2)}$. Applying the same argument to $-X$ proves the left-tail bound.

The moment-generating-function inequality and its version at $-\lambda$ implyComparing the second-order terms as $\lambda \to 0$ gives $\mathbb{E}{X}^2\le {\sigma }^2$. Since $\mathbb{E}X=0$,

A centered $Z$ is Sub-Gamma random variable in the right tail with variance factor $v$ and scale factor $c$ whenThe corresponding Bernstein's inequality is

A standard squared-sub-Gaussian lemma, obtained by integrating the sub-Gaussian tail or expanding exponential moments, states thatScaling a sub-Gamma variable by $a\ge 0$ multiplies its variance factor by $a^2$ and its scale by $a$; independent sums add variance factors and take the largest scale. DecomposeTheir sum is therefore sub-Gamma on the right with variance factorand scale factorApply Bernstein to the sum at threshold $nx$ to obtain

A kernel for density estimation is an integrable function $K$ with $\int K=1$. The kernel density estimator isIt has order $\ell$ when $\int u^{j}K(u)du=0$ for $1\le j<\ell$ and $\int |u{|}^{\ell }|K(u)|du<\infty$.

Differentiation gives the derivative kernel density estimatorDropping the negative square of its mean from the variance and using Tonelli and the substitution $u=(x-X_1)/h$,Thus $\alpha =3$ and $C_1(K)=\Vert K^{\prime }{\Vert }_2^2$.

For $r=\lfloor \beta \rfloor$, the Nikolsky class $\mathcal{N}(\beta ,L)$ consists of functions $g$ with square-integrable derivatives through order $r$ andusing the equivalent finite-difference definition when $\beta$ is an integer. Integration by parts showsTaylor expansion in $L^2$, cancellation of the moments through order $\ell -1$ for $\ell =\lceil \beta \rceil$, and the generalized Minkowski inequality giveConsequentlywhere one admissible choice isHence $\gamma =2\beta$.

The mean integrated squared error isthe sum of integrated variance and squared bias. The preceding bounds giveTaking $h\asymp n^{-1/(2\beta +3)}$ proves the MISE rate for derivative kernel density estimationThus $\delta =2\beta /(2\beta +3)$. Estimating a density of the same smoothness has variance order $(nh{)}^{-1}$ and rate $n^{-2\beta /(2\beta +1)}$; estimating its derivative is harder because differentiation amplifies high-frequency noise, changing $h^{-1}$ to $h^{-3}$.

At a target $x$, the degree-$p$ local polynomial regression estimate minimizesand ${\hat{m}}_n(x;p,h,K)={\hat{b}}_0$. Let $R_x$ have row
$(1,x_i-x,\dots ,(x_i-x{)}^p)$ and let $W_x$ be diagonal with the kernel weights. Assume the local polynomial Gram matrix $R_x^T{W}_x{R}_x$ is positive definite. The weighted normal equations giveso it is a linear estimator in nonparametric regression.

DefineInverting the two-by-two Gram matrix for $p=1$ yields

Now put $M=\min \{n,\lfloor nh\rfloor \}$. At $x=0$ and for the uniform kernel,For the usual bandwidth range $0<h\le 1$, $M=\lfloor nh\rfloor$. The elementary power-sum formulas and $|M-nh|\le 1$ show, for $r=0,1,2,3$, thatThe limiting moment determinant isFor $nh\ge 32$, the displayed errors can consume at most a fixed fraction of this value, sofor a universal $c>0$ and $0<h\le 1$.

As printed, the claim “for all $h>0$” cannot hold for this design: if $h>1$, all $n$ points receive weight $1/2$, each $s_r=O(h^{-1})$, and the determinant is $O(h^{-2})$, not bounded below by a positive multiple of $h^2$. The standard bandwidth restriction $h\le 1$ is therefore necessary for that intermediate assertion. The final bias bound remains harmless for $h>1$, since $h^2\ge 1$ and the finite design response means are uniformly bounded.

The polynomial reproduction property of local polynomial regression makes the local-linear weights reproduce both $1$ and $u$. Writefor $0\le x_i\le \min (h,1)$. The reproduced terms have zero bias. Using $s_r=O(h^r)$, the numerator contributed by the remainders is bounded byDivision by the determinant lower bound givesThe degree-zero Nadaraya–Watson estimator reproduces constants but not linear functions. The term $x_i$ therefore remains, and its boundary bias is bounded by $Ch$; this first-order boundary bias is the improvement that local linear fitting removes.

The total variation distance isIf $P\ll Q$, the Kullback-Leibler divergence isPinsker's inequality states

The squared-loss Le Cam two-point lemma says that for two experiments $P_0,P_1$ with scalar parameters ${\theta }_0,{\theta }_1$,Indeed, classify the data as $0$ when $\hat{\theta }$ is closer to ${\theta }_0$ and as $1$ otherwise. On a classification error, the estimation error is at least $|{\theta }_1-{\theta }_0|/2$. The sum of the two testing error probabilities is at least $1-\mathrm{TV}(P_0,P_1)$. Averaging the two risks and then bounding their maximum proves the result.

Let $r=\lfloor \beta \rfloor$. The Hölder class $\mathcal{H}(\beta ,L)$ on $[0,1]$ consists of functions with derivatives through order $r$ whose $r$th derivative is Hölder of order $\beta -r$ with constant $L$, with the standard integer-order convention.

Fix any $x_0\in [0,1]$. First compare the constant regression functions $m_0=0$ and $m_1=a$ with $a=c_0{n}^{-1/2}$. Both belong to every Hölder class under the seminorm convention, and the normal-product divergence isPinsker and Le Cam therefore give a lower bound $c/n$.

For the smoothness-dependent term, use the supplied smooth bump $\psi (u)=e^{-1/(1-u^2)}{1}_{\{|u|\le 1\}}$, translated one-sidedly near a boundary when necessary, and compareChoose its fixed normalization so that $m_1\in \mathcal{H}(\beta ,L)$ whenever $a\le c_{1}L{h}^{\beta }$. The Gaussian divergence satisfiesTakewith the bandwidth and amplitude truncated at constants when this expression leaves $[0,1]$. Then the divergence remains bounded andThe same construction can be placed at every $x_0$, with a one-sided bump at the endpoints. Pinsker and Le Cam, combined with the constant alternatives, provewhere $c>0$ depends only on $\beta$.