Past exam of the mathematics course of the University of Cambridge / 2014 / iii / Paper 34 (Nonparametric Statistical Theory)

Put $Q(u)=\inf \{x\in \mathbb{R}:F(x)\ge u\}$ for $0<u<1$. This is the quantile function, rather than an ordinary inverse requiring strict increase. The limits of the distribution function at infinity make $Q(u)$ finite, and the fact that $F$ is a right-continuous function ensures $F(Q(u))\ge u$. ConsequentlyIncreasing $u$ shrinks the set in the infimum, so the quantile function is a monotone function. To prove the left continuity of the quantile function, fix $u$ and let $L={\sup }_{v<u}Q(v)\le Q(u)$. If $L<Q(u)$, choose $L<y<Q(u)$. Then $F(y)<u$, so some $v\in (F(y),u)$ satisfies $Q(v)>y$, a contradiction. Thus $Q(v)\uparrow Q(u)$ as $v\uparrow u$. At a flat stretch of the distribution function, the quantile function may jump immediately to the right, consistent with this left-continuous convention.

For the uniform order statistic, count the observations at most $u$. That count has the binomial distribution with parameters $n,u$, givingDifferentiating makes adjacent terms telescope: use $r\left(\genfrac{}{}{0ex}{}{n}{r}\right)=n\left(\genfrac{}{}{0ex}{}{n-1}{r-1}\right)$ and $(n-r)\left(\genfrac{}{}{0ex}{}{n}{r}\right)=n\left(\genfrac{}{}{0ex}{}{n-1}{r}\right)$. The probability density function is thereforeIn particular, $U_{(j)}$ has the Beta distribution with parameters $j,n-j+1$.

Write $m=Q(1/2)$ for the specified median. Continuity of the distribution function gives $F(m)=1/2$ and no mass at $m$, even if the distribution function has a flat stretch there. Thus $B={\sum }_i{1}_{\{X_i<m\}}$ has the binomial distribution with parameters $n,1/2$. Except on a null event, the order-statistic confidence interval for a median covers $m$ exactly when $j\le B\le n-j$. Symmetry of the binomial distribution makes its two failure probabilities equal. Alternatively, the probability integral transform givesHence the coverage is exactly $1-\alpha$, withThe asymmetric open/closed endpoint convention does not change the coverage because the continuous probability distribution assigns no mass to the median.

For independent and identically distributed random variables sampled from a probability density function $f$, a kernel density estimator with kernel for density estimation $K$ and smoothing bandwidth $h>0$ isThe usual kernel for density estimation is nonnegative; finite $\int K^2$ makes the integrated variance finite. The mean integrated squared error and its bias-variance decomposition of mean squared error areThe interchange is justified by the Tonelli theorem.

For the exponential distribution and the specified unit-width uniform kernel, put $g_h(x)=\mathbb{E}{\hat{f}}_h(x)$. The expected value is $h^{-1}\mathbb{P}\{x-h/2\le X\le x+h/2\}$. Intersecting this interval with the positive half-line givesOn $[-h/4,0)$ the true probability density function vanishes, whereasHere $1-e^{-z}\ge z/2$ for $0\le z\le 1/4$. This leakage across the support boundary proves the integrated squared bias from a density jump:In particular, $h_0=1$ and $c=1/256$ suffice. The phenomenon differs from a smooth interior bias of a kernel density estimator: the order-one boundary error persists over a region of width proportional to $h$.

By independence, $\mathrm{Var}({\hat{f}}_h(x))=n^{-1}\{\mathbb{E}{K}_h(x-X{)}^2-g_h(x{)}^2\}$. Integrating the first term and using $\int K^2=1$ yieldsAlso $I(h)\le \int f^2=1/2$: the kernel density estimator mean is an average of translates of $f$, so the Jensen inequality followed by the Tonelli theorem bounds its squared integral.

To optimize over every $h>0$, we must establish that useful smoothing bandwidths approach zero, rather than optimize only a formal small-$h$ expression. An exact calculation supplies that justification. The autocorrelation integral of the exponential distribution density isAveraging over the two uniform windows, or integrating the piecewise expression for $g_h$, givesThese formulas show that $B$ is continuous on $(0,\infty )$, is strictly positive there because $g_h>0$ on $(-h/2,0)$, and tends to $1/2$ as $h\to \infty$. Consequently, $B$ is bounded away from zero on $[\delta ,\infty )$ for every $\delta >0$. Its expansion agrees with the supplied $B(h)=h/12+o(h)$.

The choice $h_n=\sqrt{12/n}$ givesChoose smoothing bandwidths within $1/n$ of the infimum. Their mean integrated squared error tends to zero, and nonnegativity of the integrated variance gives $B(h_n)\to 0$, hence $h_n\to 0$. For any $\epsilon \in (0,1/12)$, eventuallyThe last step is the arithmetic-geometric mean inequality. Taking the lower limit and then $\epsilon \downarrow 0$ matches the upper bound. ThusThe original PDF has this constant; the TeX aid's $1/\sqrt{2}$ is a transcription error.

For fixed-design nonparametric regression, let $x_i=i/n$ and $K_h(u)=h^{-1}K(u/h)$. The degree-$p$ local polynomial estimator takes the fitted intercept ${\hat{m}}_h(x;p)={\hat{\beta }}_0$, whereUse a nonnegative regression kernel and an invertible local polynomial Gram matrix to obtain a unique fit. For $p=0$, solving the single weighted least squares normal equation gives the local constant estimator:At the interior point $x_0$, use the standard unit-integral regression kernel convention ${\mu }_0(K)=1$. Otherwise the leading variance below is $R(K)/(nh{\mu }_0(K{)}^2)$; unlike a kernel density estimator, the local constant estimator itself is unchanged by rescaling $K$.

The local constant estimator is a linear estimator in nonparametric regression with weights $W_i=K_h(x_i-x_0)/(n{s}_{0,h})$. The independent unit-variance errors implyTo control the bias of an estimator, differentiability at this one point is enough. WriteSymmetry makes ${\mu }_1(K)=0$, and the supplied moment approximation gives $s_{1,h}=o(h)$. Therefore the linear contribution to the bias of an estimator is $m^{\prime }(x_0)s_{1,h}/s_{0,h}=o(h)$. The remaining contribution has absolute value at most $\eta (h)h$, because only $|x_i-x_0|\le h$ receive positive weight. This is the interior first-order bias cancellation for local constant regression, and gives squared bias of an estimator $o(h^2)$. Combining with the variance provesFor fixed $\epsilon >0$, take $h_{\epsilon }={\epsilon }^{-1/3}{n}^{-1/3}$. It satisfies both asymptotic smoothing bandwidth conditions, and the displayed remainder, multiplied by $n^{2/3}$, tends to zero for this fixed $\epsilon$. ThusLetting $\epsilon \downarrow 0$ proves the optimized scaled pointwise error tends to zero. The order of these limits matters: no remainder uniform in $\epsilon$ has been assumed.

For the lower bound over the whole function class, the absence of a uniform bound on derivatives permits a particularly direct argument. It also avoids assuming a normal distribution for errors when the model only supplies their first two moments. Fix $n$ and $x_0$. Choose a smooth bump function $b$ on the interval, with $b(x_0)=1$, whose support contains no design point except possibly $x_0$. Such a bump exists because the design is finite; at an endpoint use the restriction of a smooth bump on the real line. Every $m_a=ab$, $a\in \mathbb{R}$, belongs to the differentiability class.

If $x_0$ is not a design point, all observation means are zero for every $a$. The Le Cam two-point lemma applied to $a=\pm A$ has total variation distance zero and gives a lower bound tending to infinity with $A$. In fact, the supremum of the mean squared error is infinite at every such $x_0$.

If $x_0$ is a design point, only its response depends on $a$. The resulting location family is the single observation $Y=a+ϵ$; all remaining observations are independent of $a$ and give no extra information. The single-observation location minimax bound isHere is a proof covering discrete as well as continuous errors. Give $a$ the prior distribution $U[-A,A]$, independently of the noise. Put $p_M=\mathbb{P}(|ϵ|\le M)$ and $v_M=\mathrm{Var}(ϵ\mid |ϵ|\le M)$. Consider an easier experiment in which an oracle also reveals whether $|ϵ|\le M$, and reveals $a$ itself on the complementary event. On the event $|ϵ|\le M$ and $|Y|\le A-M$, every possible truncated noise value places $a=Y-ϵ$ within $[-A,A]$. The flat prior distribution then makes the Bayesian posterior noise law exactly its truncated original law. The smallest conditional squared-error loss is its conditional variance $v_M$, attained by the posterior mean.

The probability of that event is at least $p_M(1-2M/A)$ for $A>2M$, by restricting further to $|a|\le A-2M$. Thus every rule's supremum risk function is at least the original Bayes risk, which is at least the easier experiment's Bayes risk, and therefore at least $p_M(1-2M/A)v_M$. Let $A\to \infty$ and then $M\to \infty$. Finite second moment and zero mean give $p_M{v}_M\to 1$. The rule $T(Y)=Y$ has constant mean squared error one, proving the equality.

Consequently, at every $x_0\in [0,1]$, the unrestricted differentiability class satisfies the stronger conclusionThus $c=1$ works for every $n\ge 1$. There is no conflict with the preceding fixed-function limit: the worst functions, including arbitrarily narrow bumps, can vary with $n$. This is the pointwise versus uniform risk distinction. A smoothness ball with a common derivative bound would pose a different minimax risk problem.

A non-degenerate probability distribution is not concentrated at a single point. In extreme value theory, a non-degenerate distribution function $F$ is max-stable if, for each integer $n\ge 1$, constants $a_n>0,b_n\in \mathbb{R}$ satisfy $F(a_{n}x+b_n{)}^n=F(x)$ for every $x$. This says that a normalized sample maximum has the same law as one observation. A distribution function $F$ belongs to the maximum domain of attraction of a non-degenerate $G$ if there are $a_n>0,b_n$ such thatat every continuity point of $G$. Equivalently, $(M_n-b_n)/a_n$ has convergence in distribution to $G$.

The extremal types theorem says that any such non-degenerate limit is max-stable and, up to a positive affine change of variable, is exactly one of the following:These are respectively the Gumbel distribution, Fréchet distribution, and negative Weibull distribution. The theorem is also known as the Fisher–Tippett–Gnedenko theorem.

Useful sufficient conditions can be expressed using the survival function $\overline{F}=1-F$ and the right endpoint of a distribution $x_F$. An infinite endpoint with $\overline{F}(tx)/\overline{F}(t)\to x^{-\alpha }$ for every $x>0$ gives the Fréchet distribution domain. A finite endpoint with $\overline{F}(x_F-sx)/\overline{F}(x_F-s)\to x^{\alpha }$ as $s\downarrow 0$ gives the negative Weibull distribution domain. For the Gumbel distribution, it suffices that a positive Gumbel auxiliary function $a(t)$ gives $\overline{F}(t+a(t)x)/\overline{F}(t)\to e^{-x}$ for every real $x$ as $t\uparrow x_F$. These are regular variation and exponential tail-ratio conditions; they need no proof here.

In case (i), ${\overline{F}}_1(x)=(1-x{)}^2$ on $0<x<1$, so the finite-endpoint ratio is exactly $x^2$. Therefore the domain is negative Weibull with shape $2$. With $a_n=n^{-1/2}$ and $b_n=1$,and the limit is one for $x\ge 0$.

In case (ii), ${\overline{F}}_2(t)=e^{-\lambda t}$ has exponential tail ratio with the constant Gumbel auxiliary function $1/\lambda$. Therefore the domain is Gumbel. Choosing $a_n=1/\lambda$ and $b_n=\log n/\lambda$ givesIn case (iii), ${\overline{F}}_3(t)=(1+t{)}^{-\lambda }$ has regular variation of index $-\lambda$, so the domain is Fréchet with shape $\lambda$. The convenient choices $a_n=n^{1/\lambda }$ and $b_n=-1$ givewith limit zero for $x\le 0$.

Finally, define the empirical distribution function $F_n(x)=n^{-1}{\sum }_{i=1}^n{1}_{\{X_i\le x\}}$. On a sample with positive threshold $t=X_{(n-k)}$ and exactly $k$ observations strictly above $t$, its survival function has $1-F_n(t)=k/n$. Using the Tonelli theorem to integrate the finite nonnegative sum givesThis is the Hill estimator as an empirical plug-in version of the tail-integral limit. It does not prove consistency for fixed $k$; that is a separate issue.

There is a finite-sample qualification because the question permits ties and does not assume positive observations. The formula requires $1\le k<n$ and $t>0$. If $r=\#\{i:X_i>t\}<k$ because the threshold is tied, direct substitution giveswhere terms equal to the threshold contribute zero. This is generally $k/r$ times the displayed Hill estimator, rather than the same estimator. If $r=0$, the empirical plug-in denominator is zero. For a continuous probability distribution the no-tie condition holds almost surely; for a Fréchet distribution domain and fixed $k$, the threshold is positive with probability tending to one. Thus the stated plug-in identity holds at a positive untied threshold, and the literal claim for all samples from an arbitrary distribution function needs this qualification.