Past exam of the mathematics course of the University of Cambridge / 2013 / iii / Paper 33 (Nonparametric Statistical Theory)

Write $Ph=\mathbb{E}h(X)$ and $P_{n}h=n^{-1}{\sum }_{i=1}^{n}h(X_i)$. A function bracket $[\ell ,u]$ contains the measurable functions $h$ satisfying $\ell (z)\le h(z)\le u(z)$ for every $z\in T$. Require its endpoints to be integrable and call $P(u-\ell )$ its $L^1(P)$ width. A sufficient condition is that, for every $\epsilon >0$, finitely many brackets of width at most $\epsilon$ cover the whole class $\mathcal{H}$. Under this condition the uniform strong law from finite L1 bracketing statesThe conclusion holds outside a common measurable null set. If the supremum is not initially known to be measurable, this formulation means pathwise convergence on a measurable probability-one event; a pointwise separable function class, including the application below, has a measurable supremum. Pointwise brackets also ensure the sample inequalities hold simultaneously over the class.

To prove the result, choose a finite $\epsilon$-cover $[{\ell }_r,u_r]$, $1\le r\le N$. If $h$ belongs to bracket $r$, monotonicity of the empirical measure and of expectation givesandConsequentlyApply the strong law of large numbers to these finitely many integrable endpoints. On a probability-one event, the maximum tends to zero. Repeat with $\epsilon =1/q$, $q\in \mathbb{N}$, and intersect the countably many probability-one events. The limiting supremum is bounded by $1/q$ for every $q$, hence is zero. This proves the uniform law of large numbers without a boundedness assumption on the class itself.

For the moment-generating function, use the empirical measure estimatorIf $t=0$, this estimator and $m$ are both identically one. Otherwise, $X\ge 0$ makes $e^{sX}$ increasing in $s$, and $e^{sX}\le e^{tX}$ provides an integrable envelope. The dominated convergence theorem shows that $m$ is continuous on $[0,t]$, hence uniformly continuous.

For any $\epsilon >0$, choose a partition $0=s_0<s_1<\cdots <s_N=t$ so that $m(s_r)-m(s_{r-1})\le \epsilon$ for every $r$. If $s\in [s_{r-1},s_r]$, then $e^{s_{r-1}z}\le e^{sz}\le e^{s_{r}z}$ for every $z\ge 0$. These endpoint functions form finitely many integrable function brackets with the required $L^1(P)$ widths. The just-proved uniform law of large numbers therefore givesBoth functions of $s$ are continuous; their supremum equals the supremum over a countable dense subset, so it is measurable. This proves uniform consistency of an empirical moment-generating function using only the observed sample.

For the unit-width box kernel, the convolution and kernel density estimator areandIn particular $\mathbb{E}{\hat{f}}_{n,h}^K=g_h$. The scaling gives $\Vert K_h{\Vert }_2^2=h^{-1}\Vert K{\Vert }_2^2=h^{-1}$. Even if $f$ is not square-integrable, Young's convolution inequality gives $g_h\in L^2$ because $f\in L^1$ and $K_h\in L^2$.

Using independence to eliminate the cross terms in the centred estimator, and Tonelli theorem to integrate the nonnegative variance, gives the exact integrated variance of a kernel density estimator:The Cauchy-Schwarz inequality now yieldsThis constant comes from the unit $L^2$ norm of the unscaled unit-width box kernel.

For the piecewise constant probability density function, integrability forces the constants on both unbounded outer intervals to be zero. Thus $f$ is bounded, has compact support, and has finitely many jumps. Let ${\Delta }_r=f(x_r+)-f(x_r-)$ denote the jump at $x_r$. For $h$ smaller than the least gap between consecutive breakpoints, the smoothing regions $[x_r-h/2,x_r+h/2]$ do not overlap. Outside these regions the convolution equals $f$.

Within a region, put $v=x-x_r$. For $-h/2<v<0$, the bias is ${\Delta }_r(1/2+v/h)$; for $0<v<h/2$, it is $-{\Delta }_r(1/2-v/h)$. Endpoint values do not affect an $L^2$ norm. Integrating these two triangular errors givesThis is the box-kernel bias of a piecewise constant density. By the triangle inequality, with $C_f=({\sum }_r{\Delta }_r^2/12{)}^{1/2}$,Balancing this bias-variance tradeoff with $h_n=n^{-1/2}$ gives, for all sufficiently large $n$,

Use half-open intervals and setThe Haar scaling functions and Haar wavelets areFor fixed $j$, each scaling function has squared integral $2^j{2}^{-j}=1$, and different $k$ have disjoint supports, proving that this family is orthonormal. Each wavelet also has norm one and integral zero. At a fixed level, distinct wavelets have disjoint supports. At different levels, their dyadic supports are either disjoint or the finer support lies in a half on which the coarser wavelet is constant. The finer wavelet has integral zero, so their inner product vanishes. Finally, a wavelet with $l\ge 0$ is supported inside a unit interval; it is orthogonal to its containing ${\phi }_{0,k}$ because its integral is zero, and to all other unit scaling functions because their supports are disjoint. Thus both the scaling family at each fixed level and the complete stated Haar family are orthonormal sets.

The Haar refinement identity isIt shows that the closed scaling spaces $V_j$ satisfy $V_{j+1}=V_j\oplus W_j$, where $W_j$ is the closed span of the level-$j$ wavelets. The union of the $V_j$ is dense in $L^2(\mathbb{R})$: continuous compactly supported functions can be approximated in $L^2$ by their dyadic cell averages, and such continuous functions are dense in $L^2$. Hence the orthonormal Haar family is also a complete orthonormal basis.

For a locally integrable function, define the compact-support coefficient integralsThese exist without requiring $f\in L^2$. The two formulas for the Haar approximation areFor $j=0$ the wavelet sum is empty. For every fixed $j$, these sums are locally finite, so they make sense pointwise and locally in $L^1$. The refinement identities and their orthogonal two-by-two coefficient transformation establish equality by induction, even for merely locally integrable $f$. If $f\in L^2$, this is also the orthogonal projection onto $V_j$.

In particular, for $\delta =2^{-j}$ and $x\in I_k=[k\delta ,(k+1)\delta )$,Now impose the symmetry and monotonicity conditions. The limit zero and decrease on $[0,\infty )$ imply $0\le f(x)\le f(0)$ for $x\ge 0$. For $k\ge 0$, the cell average and every value inside $I_k$ lie between $f((k+1)\delta )$ and $f(k\delta )$. ThereforeSumming over the positive half-line telescopes and gives a bound $\delta f(0)$. Reflection maps each dyadic cell to another dyadic cell up to endpoints. Since $f$ is even, its Haar approximation is even almost everywhere as well, so the negative half-line has the same error. ConsequentlyThe argument proves integrability of the difference, even if $f$ itself is not integrable on the whole line. It is the symmetric monotone case of the Haar approximation error for a function of bounded variation.

For the unit-width box kernel, the order-zero local polynomial estimator minimizes, over constants $a$,A common factor such as $h^{-1}$ in these weights does not change the minimizer. Put $I_x=\{i:|i/n-x|\le h/2\}$ and $N_x=|I_x|$. If $N_x>0$, differentiating the quadratic gives its unique minimizer:The last expression is the Nadaraya–Watson estimator. Every $x\in [0,1]$ is within $1/n$ of some design point $i/n$. Since $nh\ge 2$ gives $h/2\ge 1/n$, that point is in the closed kernel window. Thus $N_x\ge 1$ and the equality holds throughout the domain, including $x=0$ and $x=1$.

For the error bound, first take the usual bandwidth range $0<h\le 1$. The clipped window $[x-h/2,x+h/2]\cap [0,1]$ has length at least $h/2$. Any closed interval of length $L$ in $[0,1]$ contains at least $\lfloor nL\rfloor$ points of the design $\{1/n,\dots ,1\}$. Consequentlywhere the last inequality uses $nh/2\ge 1$. This is a window occupancy for an equally spaced regression design bound that remains valid at the boundary.

Let $L_m=\Vert m^{\prime }{\Vert }_{\infty }$, the Lipschitz constant supplied by the bounded-derivative assumption. The bias is bounded byIndependence and the variance bound giveBy the Cauchy-Schwarz inequality and the triangle inequality,Thus $\kappa =2$ works uniformly in $x$ and the design size in this bandwidth range. The same clipped-length proof in fact works for $0<h\le 2$.

There is a genuine omission in the unrestricted formulation: an upper restriction on bandwidth is necessary for the displayed $1/\sqrt{nh}$ variance scale. Take $n=1$, $m\equiv 0$, $Y_1\sim N(0,{\sigma }^2)$ and $h\ge 2$. The window contains the sole observation for every $x\in [0,1]$, and $nh\ge 2$ holds. Its mean absolute error is $\sigma \sqrt{2/\pi }$, whereas the proposed variance term is $\kappa \sigma /\sqrt{h}$. No fixed $\kappa$ can satisfy this for unbounded $h$.

A valid statement for every $h>0$ satisfying $nh\ge 2$ uses $q=\min (h,1)$. The clipped window has length at least $q/2$, so $N_x\ge nq/2-1$, while the preceding coverage argument gives $N_x\ge 1$. If $nq\ge 4$, the first bound gives $N_x\ge nq/4$; if $nq<4$, the second does. Therefore the same bias and variance calculation proves the unrestricted correctionThis mean absolute error of local constant regression bound reduces to the requested rate for ordinary small bandwidths and saturates its stochastic term when the window covers the whole design.