Past exam of the mathematics course of the University of Cambridge / 2010 / iii / Paper 31 (Nonparametric Statistical Theory)

The empirical distribution function isIf $W$ is a standard Brownian motion on $[0,1]$, the standard Brownian bridge is $\mathbb{G}(u)=W(u)-uW(1)$. It is a centred Gaussian process with continuous paths, endpoints zero, and covarianceThe F-Brownian bridge is the time-changed Gaussian process ${\mathbb{G}}_F(t)=\mathbb{G}(F(t))$. Its covariance is $F(\min (s,t))-F(s)F(t)$, exactly the covariance of the empirical distribution process ${\alpha }_n=\sqrt{n}(F_n-F)$.

The Donsker theorem for empirical distribution functions states that for every cumulative distribution function $F$,Thus this is a function-space central limit theorem, not merely convergence of each finite collection of evaluations. The limit is a tight centred Gaussian process. No continuity of $F$ is required for this statement; a jump of $F$ can give a jump of ${\mathbb{G}}_F$.

For precision, ${\ell }^{\infty }(\mathbb{R})$ is generally nonseparable, and the finite-sample process need not be a Borel-measurable random element for its norm topology. The displayed convergence in distribution has the usual empirical-process interpretation: for every bounded continuous real functional $H$, the outer expectation and inner expectation of $H({\alpha }_n)$ converge to $\mathbb{E}H({\mathbb{G}}_F)$. The outer expectation is the infimum of expectations of measurable majorants, and the inner expectation is its negative applied to $-H({\alpha }_n)$. For measurable real statistics this gives ordinary convergence in distribution. The limit is tight and separably supported: $g\mapsto g\circ F$ maps the separable space of continuous bridge paths continuously into ${\ell }^{\infty }(\mathbb{R})$.

The map $z\mapsto \Vert z{\Vert }_{\infty }$ is Lipschitz, sinceConsequently the continuous mapping theorem applied to the Donsker theorem for empirical distribution functions gives $\Vert {\alpha }_n{\Vert }_{\infty }\stackrel{d}{\to }\Vert {\mathbb{G}}_F{\Vert }_{\infty }$. The statistic on the left is measurable: both $F_n$ and $F$ are right-continuous, so its supremum equals the supremum over rational $t$.

If $F$ is continuous, its limits at $-\infty$ and $+\infty$ are zero and one. The intermediate value theorem therefore shows that $F(\mathbb{R})$ contains $(0,1)$, even when $F$ is not strictly increasing. A continuous Brownian bridge is zero at its endpoints, so pathwiseWe obtain the Kolmogorov-Smirnov theorem in the required form:Continuity of $F$ is what makes the limit independent of the unknown cumulative distribution function.

Choose $c_{\alpha }$ with $\mathbb{P}(Z\le c_{\alpha })=1-\alpha$. The law of $Z$ has no atoms. For example, the Brownian reflection principle and conditioning a Brownian motion on its endpoint give $\mathbb{P}(\max \mathbb{G}>a)=e^{-2a^2}$ for $a>0$: the reflected endpoint density at zero is the normal density at $2a$, divided by its value at zero. Hence both one-sided maxima have continuous laws, and an atom of $Z$ would require an atom of one of them. Also $\mathbb{P}(Z=0)=0$, since $\mathbb{G}(1/2)$ is nondegenerate. Thus such quantiles give the desired limiting coverage.

An asymptotic simultaneous confidence band isIndeed, because $0\le F\le 1$, the event that $L_n(t)\le F(t)\le U_n(t)$ for every $t$ is precisely $\sqrt{n}\Vert F_n-F{\Vert }_{\infty }\le c_{\alpha }$. Its probability tends to $1-\alpha$. This is simultaneous coverage over all real $t$, rather than separate pointwise confidence intervals.

For a kernel for density estimation $K$ and smoothing bandwidth $h>0$, the kernel density estimator isWe use the convention that a kernel of order ell hasThe kernel for density estimation is integrable and can have negative values; requiring nonnegativity would prevent its second moment from vanishing. Some conventions additionally require a nonzero moment of order $\ell$. The construction below covers that convention too.

The Legendre polynomial kernel construction uses the polynomial reproducing kernel at zero:Every polynomial $p$ of degree at most $\ell -1$ has the expansion $p={\sum }_{m=0}^{\ell -1}⟨p,{\varphi }_m⟩{\varphi }_m$, since the displayed Legendre polynomials have those successive degrees and form an orthonormal basis of this polynomial space. ThereforeTaking $p=1,u,\dots ,u^{\ell -1}$ proves all the required moment identities. Boundedness and compact support give every absolute moment and a finite L2 norm. If a nonzero moment of order $\ell$ is required, replace this kernel byThe lower moments are unchanged by orthogonality. If $a_{\ell }=⟨u^{\ell },{\varphi }_{\ell }⟩$, then $a_{\ell }\ne 0$ because otherwise $u^{\ell }$ would lie in the polynomial space of degree at most $\ell -1$. Evaluating its full Legendre expansion at zero gives $\int u^{\ell }{K}_{\ell }=-{\varphi }_{\ell }(0)a_{\ell }$, so $\int u^{\ell }{\tilde{K}}_{\ell }=a_{\ell }\ne 0$.

For instance, a simple kernel satisfying the moment conditions needed for three derivatives isIts negative wings cancel its central contribution to the second moment.

Here is the third-order pointwise kernel error bound. Write $B=\Vert f{\Vert }_{\infty }$ and $M_3=\Vert f^{(3)}{\Vert }_{\infty }$, and use any square-integrable kernel with unit integral, first two moments zero and ${\mu }_3=\int |u{|}^3|K(u)|du<\infty$. A change of variables givesBy the Taylor theorem with Lagrange remainder,The moment cancellations consequently give the uniform bias of a kernel density estimator boundFor the variance, independence and another change of variables giveThe triangle inequality and Cauchy-Schwarz inequality now yieldBalancing $h^3$ against $(nh{)}^{-1/2}$ givesThe constant is independent of both $n$ and $x$. The original PDF has exponent $3/7$; the converted TeX's $3/2$ is a transcription error.

To obtain the final parametric rate, we choose a flat-top kernel for density estimation. This kernel also satisfies all the preceding moment conditions, so a single kernel works for both constructions, with different bandwidth choices. With the Fourier transform convention $\hat{g}(t)=\int e^{-itx}g(x)dx$, choose a real even $C^{\infty }$ function $\eta$ equal to one on $[-1,1]$ and zero outside $[-2,2]$, and putSuch a cutoff is explicit: if $\rho (s)=e^{-1/s}$ for $s>0$ and zero otherwise, take $\eta (t)=\rho (2-|t|)/[\rho (2-|t|)+\rho (|t|-1)]$. It is constant near zero, and its endpoint transitions are smooth. Repeated integration by parts shows that $K$ and all its derivatives decay faster than every power, so $K$ lies in the Schwartz space. The Fourier inversion and moment differentiation of the Fourier transform giveThus this kernel can be used with $h_n=n^{-1/7}$ for the preceding general bound.

For the sharper example choose the probability density functionTo check this is a density and identify its Fourier transform, direct integration givesThe continuous value at zero is as displayed. This function is nonnegative and integrable, being bounded near zero and $O(x^{-2})$ at infinity. Fourier inversion gives $\hat{f}(t)=(1-|t|{)}_{+}$, and in particular $\int f=\hat{f}(0)=1$. Differentiation of this compact frequency integral also shows that $f$ and $f^{(3)}$ are bounded. Since $\eta =1$ on the support of $\hat{f}$, the convolution theorem givesBoth sides are continuous, so the equality holds at every $x$. There is therefore exactly zero bias of a kernel density estimator at bandwidth one, and the variance bound above givesThis is parametric-rate kernel estimation of a bandlimited density. A fixed bandwidth is appropriate because it reproduces this particular density exactly; a generic density would retain a nonzero smoothing bias.

For an orthonormal wavelet construction, setThe scaling function $\varphi$ generates the approximation spaces $V_j$, while $\psi$ generates the detail spaces $W_j$ in the multiresolution analysis, with $V_{j+1}=V_j\oplus W_j$. At coarse resolution zero, the family $\{{\varphi }_{0,k}{\}}_k\cup \{{\psi }_{j,k}{\}}_{j\ge 0,k}$ is an orthonormal basis of $L^2(\mathbb{R})$.

For the given square-integrable function the coefficients and wavelet series areFor real wavelets these are the real L2 inner products; use complex conjugates for a complex basis. The series converges in the L2 norm, and its coefficients satisfy the Parseval identity for a Hilbertian basisAn $L^2$ hypothesis alone does not assert pointwise convergence.

The resolution-$J$ wavelet projection is the orthogonal projection onto $V_J$:The two expressions coincide by iterating $V_{j+1}=V_j\oplus W_j$. Orthogonality gives $\Vert m-P_{J}m{\Vert }_2^2={\sum }_{j\ge J,k}|b_{j,k}{|}^2$, which tends to zero. One can equivalently write the projection with kernel $A_J(x,y)={\sum }_k{\varphi }_{J,k}(x){\varphi }_{J,k}(y)$, interpreted as the same orthogonal projection.

In fixed-design nonparametric regression, the observations satisfywith independent errors. In the usual Gaussian version ${\epsilon }_i\sim N(0,{\sigma }^2)$. The regression function is evaluated at the design sites, so it must be specified pointwise there, not only as an $L^2$ equivalence class. For $x_i=i/n$, the observations concern $m$ on $[0,1]$. They cannot identify its values outside this interval. Use an interval-adapted wavelet basis on $[0,1]$, consisting of coarse functions ${\varphi }_{0,k}$ and detail functions ${\psi }_{j,k}$; a Haar wavelet basis is one explicit choice.

Here is a wavelet regression estimator valid for every $n$. Divide $[0,1]$ into cells $I_i=((i-1)/n,i/n]$, with the value at zero assigned arbitrarily, and form the observed step function ${\tilde{Y}}_n(t)=Y_i$ on $I_i$. For any normalized basis function $b_{\lambda }$, estimate its coefficient byThe wavelet regression estimator at resolution $J=J_n$ is thereforeOnly finitely many interval basis functions occur. Its expectation is $P_J{m}_n^{\mathrm{grid}}$, where $m_n^{\mathrm{grid}}$ equals $m(i/n)$ on $I_i$. Thus approximation of $m$ includes the resolution bias and the grid discretization bias. For continuous $m$, the latter tends uniformly to zero by uniform continuity. No smoothness or consistency claim for arbitrary pointwise representatives of $L^2$ functions is needed to define the estimator.

The common point-evaluation version replaces each weight by $b_{\lambda }(i/n)/n$, givingThese are Riemann-sum approximations to the coefficient integrals. For Haar wavelets, $n=2^L$, and $J\le L$, they equal the integrated-weight coefficients exactly if the dyadic cells and pointwise basis versions are chosen right-closed. Each retained basis function is then constant on every sampling cell. This gives a concrete implementation by the discrete wavelet transform. A sensible finest resolution has $2^J\le n$; beyond the grid scale there are details the data cannot resolve.

For wavelet coefficient thresholding, retain the coarse coefficients but discard detail coefficients dominated by noise. A hard-threshold version isAlternatively, soft thresholding replaces each detail coefficient by $\mathrm{sgn}(\hat{b})(|\hat{b}|-\tau {)}_{+}$. The unthresholded wavelet regression estimator retains all details at the chosen resolution; thresholding retains evidence for local features while suppressing small fluctuations.

The thresholds can be derived from a simultaneous wavelet coefficient noise bound. Write $q_{\lambda }^2={\sum }_i{w}_{\lambda i}^2$. Independence giveswhere the inequality applies the Cauchy-Schwarz inequality on each cell of length $1/n$. In the dyadic Haar implementation, $q_{\lambda }^2=1/n$ exactly. For Gaussian errors, the normalized coefficient noise is standard normal. More generally, independent centred sub-Gaussian random variables with parameter $\sigma$ giveIf $N_J$ detail coefficients are considered, a union bound consequently justifiesWith probability at least $1-\delta$, every retained detail's noise is below its corresponding threshold; independence between the coefficient errors is unnecessary. With $N_J$ of order $n$ and $q_{\lambda }=n^{-1/2}$, these thresholds have the familiar order $\sigma \sqrt{\log n/n}$. Coefficients larger than their noise level represent potential signal, and a coefficient zero in the discretized mean is not selected on this simultaneous event. If $\sigma$ is unknown, a consistent noise-scale estimate can be substituted; for Gaussian errors and predominantly negligible finest-level signal, the median absolute finest-level coefficients divided by the standard-normal absolute median estimates $\sigma /\sqrt{n}$. Without a Gaussian or sub-Gaussian tail assumption, finite variance alone does not justify the logarithmic thresholds: Chebyshev inequality and the union bound instead permit ${\tau }_{\lambda }=\sigma q_{\lambda }\sqrt{N_J/\delta }$.

A real functional $\Phi$ is Hadamard differentiable at $s_0$ if there exists a continuous linear functional $L={\dot{\Phi }}_{s_0}:S\to \mathbb{R}$ such that, whenever $t_j\ne 0$, $t_j\to 0$ and $h_j\to h$ in the normed vector space,If $\Phi$ is defined only on a subset, require $s_0+t_j{h}_j$ to lie in its domain. The variation of the directions $h_j$ is essential: convergence along each fixed direction alone is a weaker differentiability condition. Tangential Hadamard differentiability restricts the limit directions to a specified subspace; here the derivative is defined on all of $S$.

We prove the functional delta method using compact sets, so do not need an unjustified application of the Skorokhod representation theorem on an arbitrary nonseparable normed vector space. As usual for function-space convergence in distribution, the limiting random element is tight, and the real statistics are measurable, or convergence is understood through outer expectations. These are automatic under the usual separable complete random-element formulation; the empirical-process formulation used below has a tight separably supported limit.

For $t\ne 0$ putHadamard differentiability says $R_{t_j}(h_j)\to 0$ for every convergent sequence of directions, since continuity gives $L(h_j)\to L(h)$. It implies a Hadamard remainder near a compact set: for every compact $K\subset S$ and $ϵ>0$ there are $a>0$ and $\delta >0$ such thatTo prove it, suppose it failed. Choose $t_j\to 0$ and $h_j$ at distance less than $1/j$ from $K$ with $|R_{t_j}(h_j)|\ge ϵ$. Choose $k_j\in K$ with $\Vert h_j-k_j\Vert <2/j$. A subsequence of $k_j$ converges by compactness, so the corresponding $h_j$ converge too. The defining derivative limit contradicts the lower bound on $R_{t_j}(h_j)$.

Now let $Z_n=r_n(X_n-s_0)\stackrel{d}{\to }X$ and $t_n=1/r_n$. Given $\eta >0$, tightness of a probability measure provides a compact $K$ with $\mathbb{P}(X\in K)>1-\eta$. For the open $\delta$-neighbourhood $K^{\delta }$, the Portmanteau theorem givesOn $K^{\delta }$ the preceding uniform remainder is less than $ϵ$ for all sufficiently large $n$. Consequently ${\mathrm{limsup}}_n\mathbb{P}(|R_{t_n}(Z_n)|\ge ϵ)\le \eta$. Since $\eta$ is arbitrary, this remainder tends to zero in probability. We have proved the actual expansionContinuity of $L$ and the continuous mapping theorem give $L(Z_n)\stackrel{d}{\to }L(X)$. The vanishing remainder and Slutsky theorem therefore yieldThis derivation uses the derivative definition, rather than assuming a uniform Fréchet remainder over every bounded set. If outer expectations are needed, the same compact-neighbourhood argument uses outer probabilities and gives the corresponding empirical-process functional delta method.

For the statistical application, use the sample's empirical distribution function and the plug-in estimatorHere the printed $L^{\infty }$ denotes bounded pointwise functions with the supremum norm, namely ${\ell }^{\infty }(\mathbb{R})$, rather than equivalence classes modulo Lebesgue null sets. Assume Hadamard differentiability at the true $F$ and the usual measurability of the scalar estimator. The Donsker theorem for empirical distribution functions and the proved functional delta method giveNo continuity of $F$ is needed for this application. Since the derivative is continuous and linear and the bridge is a centred Gaussian process, $Z_F$ is a centred Gaussian real random variable, possibly identically zero.

Finally, a finite real weak limit ensures the required boundedness in probability. Choose $M$ so large that $\mathbb{P}(|Z_F|\ge M)<ϵ/2$. Applying the closed-set part of the Portmanteau theorem to $\{z:|z|\ge M\}$ givesHence there is $n_0$ such that, for every $n\ge n_0$,In particular the upper-tail condition in the question holds. The plug-in estimator has estimation error $O_{\mathbb{P}}(n^{-1/2})$, even when its limiting variance is zero.