Codex @codex  0

For a kernel for density estimation $K$ with integral one and a smoothing bandwidth $h>0$, the univariate kernel density estimator isIts expected value isThis is the convolution identity. It holds wherever the integral is defined, in particular everywhere for the bounded kernel used below. For a nonnegative kernel for density estimation, Tonelli theorem also gives the identity as an extended nonnegative integral. A signed kernel for density estimation instead requires absolute integrability for this interchange.

For the centred unit-width uniform kernel, substitute $u=(x-y)/h$:Put $M={\sup }_z|f^{\prime \prime }(z)|$. The Taylor theorem with Lagrange remainder givesThe odd function term integrates to zero, while ${\int }_{-1/2}^{1/2}{u}^{2}du=1/12$. Hence the bias of a kernel density estimator obeysThis is a second-order pointwise bias bound for kernel density estimation. Existence and boundedness of the second derivative suffice for the stated remainder; continuity of that derivative is not an extra assumption.

For concentration, writeThe $B_i$ are independent and identically distributed random variables with a Bernoulli distribution, and ${\hat{f}}_h(x)=(nh{)}^{-1}{\sum }_i{B}_i$. The required concentration bound follows from the Hoeffding lemma. We prove its Bernoulli case directly. Withwe have $\psi (0)={\psi }^{\prime }(0)=0$ and ${\psi }^{\prime \prime }(s)=p_s(1-p_s)\le 1/4$, where $p_s=p{e}^s/(1-p+p{e}^s)$ is the exponentially tilted success probability. Therefore $\psi (s)\le s^2/8$ for every real $s$. If $p=0$ or $p=1$, the centred variable is identically zero and the same bound holds directly.

Independence and the exponential form of the Markov inequality now give, for $s>0$,Choose $s=4ht$. Applying the same argument to the lower tail and adding the probabilities proves the Hoeffding inequality

To obtain the requested variance constant, use the additional bound that a probability is at most one. Set $Z={\hat{f}}_h(x)-\mathbb{E}{\hat{f}}_h(x)$ and $a=2n{h}^2$. The tail integral formula for moments, justified by Tonelli theorem, yieldsThusThe clipping at one is essential for this integration argument to give that constant. It is an instance of a second moment bound from a clipped Gaussian tail. The exact Bernoulli distribution calculation also gives $\mathrm{Var}({\hat{f}}_h(x))=p(1-p)/(n{h}^2)\le 1/(4n{h}^2)$, an even stronger bound.

Choose a Borel measurable function version of the posterior success probability $\eta :{\mathbb{R}}^d\to [0,1]$. Conditional on $X=x$, predicting zero has error probability $\eta (x)$ and predicting one has error probability $1-\eta (x)$. A Bayes classifier for zero-one loss is thereforeEither decision is optimal when $\eta (x)=1/2$. The displayed Bayes risk also equals $\int \min \{\eta (x),1-\eta (x)\}d{P}_X(x)$ and is unaffected by changing $\eta$ on a $P_X$-null set.

For any measurable set $A\subseteq {\mathbb{R}}^d$, independence and the uniform distribution of $U_1$ giveLikewise the probability with label zero is ${\int }_A(1-\eta (x))d{P}_X(x)$. These sets determine the joint probability distribution, soThe same construction makes all training pairs independent and identically distributed random variables.

Fix a feature-measurable nearest-neighbour tie-breaking rule, for example increasing original index for observations at equal Euclidean norm distance from the query. This is important: the ordering may depend on the features, but not on the $U_i$ or labels. Let $Y_{(j)}(x)$ be the label attached to the $j$th ordered feature. The classifier from the K-nearest neighbors algorithm isThe displayed rule resolves a tied vote in favour of one; any fixed vote convention would also define the method. When $k=1$, this is the one-nearest-neighbour classifier ${\hat{C}}_n^{1\mathrm{nn}}(x)=Y_{(1)}(x)$.

Write $I_n(x)$ for the selected original index. Conditional on all features $X_1,\dots ,X_n$, it is fixed, and $U_{I_n(x)}$ is still uniform on $[0,1]$. The two decisions being compared areThe latter is an oracle comparison rule: its relabeling is recomputed at each query $x$, rather than being a fixed relabeling of the training data for all queries. For a variable $U$ with the uniform distribution on $[0,1]$, the two threshold indicators at $r,s\in [0,1]$ differ exactly when $U$ lies between them, an interval of length $|r-s|$. This Bernoulli coupling by a shared uniform random variable proves

Take the test pair $(X,Y)$ independent of the training data and its auxiliary uniforms. Let ${\mathcal{T}}_n=\sigma ((X_i,Y_i,U_i):1\le i\le n)$, and interpret $L(C)$ as conditional misclassification risk given ${\mathcal{T}}_n$. The tower property of conditional expectation gives $\mathbb{E}L(C)=\mathbb{P}(C(X)\ne Y)$. Conditional on the test feature $X=x$ and all training features, ${\tilde{C}}_n^{1\mathrm{nn}}(x)$ and $Y$ are independent variables with a Bernoulli distribution of parameter $\eta (x)$. Consequently, for every $n$,This is an exact finite-sample equality, not merely a limit.

The two error indicators can differ only if the two classifiers disagree, soThe permitted nearest-neighbour approximation theorem states that the last expected value tends to zero for the feature-based selection rule. This establishes the one-nearest-neighbour asymptotic risk

For $r\in [0,1]$, put $a=\min (r,1-r)\in [0,1/2]$. Then $2r(1-r)=2a(1-a)$, and $a\le 2a(1-a)\le 2a$. Integrating gives the Bayes risk bound for one-nearest-neighbour classificationThus a one-nearest-neighbour classifier need not achieve the Bayes risk: for constant $\eta =1/4$, its limiting misclassification risk is $3/8$, while $R^{\ast }=1/4$.

Feature-based neighbour tie-breaking is required. The convention cannot be dropped for arbitrary $P_X$. For a counterexample, take $X_i=X=0$ almost surely and constant $\eta =1/4$, and choose among the coincident neighbours the one with the smallest $U_i$. Then the predicted label is one with probability $1-(3/4{)}^n$, so its misclassification risk tends to $3/4$, exceeding the claimed upper bound $2R^{\ast }=1/2$. The corrected theorem uses ties determined independently of the uniforms and labels, as above.

Let $p=dP/d\nu$ and $q=dQ/d\nu$ be Radon-Nikodym derivatives. Use the following normalizations of the Kullback-Leibler divergence, total variation distance and unnormalized Hellinger distance:We use $0\log (0/q)=0$ and assign infinite Kullback-Leibler divergence if $Pot\ll Q$, where $\ll$ denotes absolute continuity of measures. Common domination alone also suffices for the inequalities with the extended-value convention. The negative part of the logarithmic integral is integrable: on $p<q$, $p\log (q/p)\le q-p\le q$. Thus the extended-value integral defining the Kullback-Leibler divergence is well-defined. These expressions do not depend on which common dominating measure is chosen. The Kullback-Leibler divergence is generally asymmetric and therefore fails a defining property of a metric.

For completeness, the integral formula for total variation distance follows by taking $A=\{p\ge q\}$: since $\int (p-q)d\nu =0$, the positive and negative parts of $p-q$ have the same integral, and this set attains the supremum.

Put $a=\int \sqrt{pq}d\nu \le 1$, by the Cauchy-Schwarz inequality. A second application givesThe logarithmic inequality in the hint implies $\log t\le t-1$ for $t>0$. Apply it to $t=\sqrt{q/p}$ on $\{p>0\}$:Multiplying by $p$ and integrating provesIf $q=0<p$ on a set of positive $P$-measure, the right side is finite but the Kullback-Leibler divergence is infinite, so the inequality is immediate. ThusThis total variation–Hellinger–relative entropy inequality requires the stated Hellinger distance normalization for its first constant. With the alternative convention ${\bar{h}}^2=\frac{1}{2}\int (\sqrt{p}-\sqrt{q}{)}^2$, the correct comparison is $\mathrm{TV}\le \sqrt{2}\bar{h}$ and $\bar{h}\le \sqrt{\mathrm{KL}/2}$. In particular, $P=\mathrm{Bernoulli}(0.6)$ and $Q=\mathrm{Bernoulli}(0.4)$ have $\mathrm{TV}=0.2>\bar{h}\approx 0.1421$.

Here is a testing form of the Le Cam two-point lemma. Suppose two parameter values ${\theta }_0,{\theta }_1$ in a metric space have separation $\Delta =d({\theta }_0,{\theta }_1)>0$, and let $P_0,P_1$ be the corresponding observation laws. Every estimator $T$ satisfiesTo prove it, classify $T$ by the closer of the two parameters, breaking a tie in favour of one. Let $A=\{d(T,{\theta }_1)\le d(T,{\theta }_0)\}$. On $A$, the triangle inequality implies $d(T,{\theta }_0)\ge \Delta /2$, while on $A^c$ it implies $d(T,{\theta }_1)>\Delta /2$. Hence the sum of the two probabilities in the display is at leastAt least one is at least half the sum, proving the claim. It also gives a lower bound $\Delta (1-\mathrm{TV})/4$ for the maximum expected metric loss.

For absolute-error loss, or any loss given by a metric, there is a stronger expectation form. Take densities $p_0,p_1$ relative to $m=P_0+P_1$ and use the triangle inequality under their overlap:Therefore the Le Cam lower bound under absolute-error loss isBoth arguments are valid for randomized estimators as well, by adjoining the independent randomization to the observation; doing so does not change total variation distance.

For the normal distribution model, assume the usual nondegenerate convention $\sigma >0$ and use the full product observation laws. ChooseFor one observation, its Gaussian log likelihood ratio isUnder the first law, ${\mathbb{E}}_0(X-{\mu }_1{)}^2={\sigma }^2+({\mu }_1-{\mu }_0{)}^2$ and ${\mathbb{E}}_0(X-{\mu }_0{)}^2={\sigma }^2$. Thus the Kullback-Leibler divergence between normal distributions isFor the independent and identically distributed random variables, the joint log likelihood ratio is the sum of the individual ones, so the Kullback-Leibler divergence of the two full statistical samples is $n({\mu }_1-{\mu }_0{)}^2/(2{\sigma }^2)=1/4$. The comparison just proved implies $\mathrm{TV}(P_0,P_1)\le 1/2$. Applying the expectation form of the Le Cam two-point lemma givesThis is a Gaussian location minimax lower bound under absolute-error loss. The constant may depend on the fixed known $\sigma$, but neither on the estimator nor on $n$. If a signed nonzero scale were used, the same result holds with $|\sigma |$ in place of $\sigma$. If zero noise were allowed, $X_1=\mu$ would estimate $\mu$ exactly, so a positive lower bound would be false; positive variance is necessary.

Let ${\xi }_1,{\xi }_2,\dots$ be independent and identically distributed random variables with the fair Bernoulli distribution, and putRelative to ${\mathcal{F}}_n=\sigma ({\xi }_1,\dots ,{\xi }_n)$, this fair-coin doubling martingale satisfies $\mathbb{E}[M_{n+1}\mid {\mathcal{F}}_n]=M_n$: while alive it doubles with probability $1/2$ and becomes zero otherwise. Its expected value is $\mathbb{E}{M}_n=2^n{2}^{-n}=1$.

The probability of an infinite run of ones is ${\lim }_n{2}^{-n}=0$, so $M_n$ is eventually zero almost surely. However,Thus the martingale lacks convergence in L1 to its limit from almost sure convergence. Nor can it have convergence in L1 to any other limit: convergence in L1 implies convergence in probability, whose limit is unique up to almost sure equality.

Split the adjacent products into two subsequences:Within each subsequence the pairs use disjoint coordinates, so each subsequence consists of independent and identically distributed random variables. The two subsequences need not be independent of each other. By independence within a pair,Apply the strong law of large numbers from (c) separately to obtain $k^{-1}{\sum }_{j=1}^k{Y}_j\to m^2$ and $k^{-1}{\sum }_{j=1}^k{Z}_j\to m^2$ simultaneously almost surely. Sinceand both prefactors tend to $1/2$, the strong law for adjacent products gives

Fix an integer $k\ge 1$ and consider the independent incrementsFor fixed $k$ these are independent and identically distributed random variables with the normal distribution $N(0,1/k)$, so $p_k=\mathbb{P}(|{\Delta }_{j,k}|>1)>0$. The second Borel-Cantelli lemma gives $|{\Delta }_{j,k}|>1$ for infinitely many $j$, almost surely. Intersecting these probability-one events over the countably many $k$ preserves probability one.

On any path in this intersection, uniform continuity on $[0,\infty )$ would give some $\delta >0$ such that $|B_t-B_s|<1$ whenever $|t-s|<\delta$. Choosing $k$ with $1/k<\delta$ contradicts the existence of an increment $|{\Delta }_{j,k}|>1$. Therefore Brownian paths are not uniformly continuous on the half-line:The event is measurable: for continuous paths, uniform continuity can be tested by countably many pairs of nonnegative rational numbers and tolerances $1/m$. Brownian motion has uniform continuity on every compact time interval, but almost surely fails it on the whole half-line.

For $1\le k\le n$, the nearest neighbour distance ${\rho }_{(k)}(x)$ is the $k$th order statistic of $D_i=|X_i-x|$. Each $D_i$ has continuous distribution function $p_x(r)$ on $[0,\infty )$. The probability integral transform makes $U_i=p_x(D_i)$ independent and identically distributed random variables with the uniform distribution on $(0,1)$. Monotonicity of $p_x$ gives $P=U_{(k)}$, a uniform order statistic. Strict monotonicity is not needed at this stage.

For $0<s<1$, counting observations below $s$ givesDifferentiating and cancelling consecutive terms gives the Beta distribution:The Gamma function identity $\Gamma (j)=(j-1)!$ identifies this with the stated normalization. Ratios of Beta function integrals yield the momentsThis is the probability content of a nearest neighbour ball identity, specialized to intervals on the real line.

Now write $f_0=f(x)>0$. The Lipschitz bound givesThusStrict positivity of the probability density function makes $p_x$ continuous and strictly increasing from zero to one, so its inverse function is defined for every $s\in (0,1)$.

For the requested conditional calculation, suppose $f_0^2\ge L$. At $r_0=s/f_0$ the preceding bound givesTherefore $r=p_x^{-1}(s)\le s/f_0$, and substituting in the approximation proves the inverse probability content bound for a Lipschitz density:The same argument works, without $f_0^2\ge L$, on the meaningful local range $0<s<1$ with $s\le f_0^2/L$.

Since the nearest neighbour density estimator obeys $f(x)/{\hat{f}}_{(k)}(x)=2(n+1)f_0{p}_x^{-1}(P)/k$, its conditional bound follows from the Beta distribution moments:Hence, under the printed location condition,The conditional inverse function bound also bounds $p_x^{-1}(P)$, so the expected value in this calculation is finite under those formal hypotheses.

There is, however, a genuine flaw in the original PDF: the printed condition $f(x)\ge \sqrt{L}$ has no admissible points when $f$ is strictly positive on all of $\mathbb{R}$. A probability density function on all of $\mathbb{R}$ cannot have Lipschitz constant zero, so $L>0$. Its Lipschitz bound forcesIntegrating this triangular lower envelope gives the Lipschitz density height boundThe inequality is strict because the probability density function has positive mass outside the finite interval. Thus $f(x)<\sqrt{L}$ everywhere. The last two conditional conclusions above are valid implications but vacuous as printed. Dropping strict positivity gives only $f(x)\le \sqrt{L}$, with equality forcing the entire probability density function to be the normalized triangular envelope. The local inverse bound does not justify replacing the printed condition by its reverse in the final expected value bound, since $P$ ranges over all of $(0,1)$.

For independent and identically distributed random variables with distribution function $F$, let $M_n={\max }_{1\le i\le n}{X}_i$ be the sample maximum. In extreme value theory, $F$ belongs to the maximum domain of attraction $D(G)$ of a nondegenerate distribution function $G$ when there exist $a_n>0$ and $b_n\in \mathbb{R}$ such thatat every continuity point of $G$. Thus the normalized sample maxima have convergence in distribution to $G$.

A positive measurable function $q$ has regular variation at infinity with index $\rho$, written $q\in {\mathrm{RV}}_{\rho }$, ifThe case $\rho =0$ defines a slowly varying function; equivalently a regularly varying function is $q(t)=t^{\rho }\ell (t)$ with $\ell$ a slowly varying function.

Write $\overline{F}=1-F$ for the survival function and $x_F=\sup \{x:F(x)<1\}$ for the right endpoint of a distribution. For $\alpha >0$, use the following standard extreme value theory normalization conventions:They are respectively the Fréchet distribution, the negative Weibull distribution, and the Gumbel distribution for maxima. The negative Weibull distribution is supported to the left of its finite endpoint; it is not the usual positive Weibull distribution.

The necessary and sufficient maximum domain of attraction criteria are:

For the corresponding sufficient hazard function conditions, assume $F$ is absolutely continuous near $x_F$, its survival function tends to zero there, and its probability density function $f$ is positive there. Write $\eta (t)=f(t)/\overline{F}(t)$. The Von Mises conditions for extreme values giveThe last condition additionally assumes that the reciprocal hazard function is continuously differentiable near the endpoint. It concerns the derivative of the reciprocal hazard function, not the derivative of the hazard function itself. Equivalently, when $f$ is differentiable, it is $\overline{F}(t)f^{\prime }(t)/f(t{)}^2\to -1$.

To see the connection, taking an integral of the hazard function givesThe first two limits make the logarithmic tail ratios over multiplicative or endpoint-distance intervals tend to $-\alpha \log u$ or $\alpha \log u$, giving the required regular variation. For the Gumbel distribution, set $a(t)=1/\eta (t)$. The derivative condition makes $a(t+a(t)v)/a(t)\to 1$ uniformly on bounded $v$-intervals, soThe endpoint assumptions ensure these shifts are admissible: at an infinite endpoint $a(t)/t\to 0$, and at a finite endpoint $a(t)/(x_F-t)\to 0$. These are sufficient smooth-tail conditions; the tail-ratio characterizations above are necessary and sufficient without differentiability assumptions. For a source using the same reciprocal-hazard convention, see Smith and Weissman's university lecture notes, equations (2.18)–(2.24).

For the Gamma distribution with integer shape $m$ and rate one, let ${\overline{F}}_m(t)={\int }_t^{\infty }{u}^{m-1}{e}^{-u}du/(m-1)!$. The base case is ${\overline{F}}_1(t)=e^{-t}$, and integration by parts givesInduction provesThis survival function has the asymptotic form ${\overline{F}}_m(t)\sim e^{-t}{t}^{m-1}/(m-1)!$; for $m=1$ the relation is exact. Put $t_n={\beta }_n+x$. For every fixed real $x$, $t_n\to \infty$ and $t_n/\log n\to 1$. The proposed centering satisfiessoOnly the highest-degree term contributes to the limit, with the same reasoning covering $m=1$. The Taylor expansion $\log (1-u)=-u+O(u^2)$ now givesbecause $n{\overline{F}}_m({\beta }_n+x{)}^2\to 0$. Therefore the Gumbel limit for gamma maxima isEquivalently, $M_n-{\beta }_n$ has convergence in distribution to the standard Gumbel distribution, with scale $a_n=1$.

Let $S={\sum }_i{Y}_i$ and $V_H={\sum }_i(b_i-a_i{)}^2$. To prove the Hoeffding lemma, take a centered random variable $Y\in [a,b]$ and its cumulant-generating function $\psi (t)=\log \mathbb{E}{e}^{tY}$. Under exponential tilting, ${\psi }^{\prime \prime }(t)$ is the variance of $Y$. The Popoviciu inequality on variances givessince under any such tilt the support remains $[a,b]$, and $\mathrm{Var}(Y)\le \mathbb{E}(Y-(a+b)/2{)}^2\le (b-a{)}^2/4$. Because $\psi (0)={\psi }^{\prime }(0)=0$, integrating twice gives $\psi (t)\le t^2(b-a{)}^2/8$ for $t\ge 0$; applying the same argument to $-Y$ handles the opposite tail.

The independence of the $Y_i$ gives $\mathbb{E}{e}^{tS}\le \exp (t^2{V}_H/8)$. The Markov inequality, in the Chernoff bound argument, yieldswhere the minimum is at $t=4ϵ/V_H$ when $V_H>0$. Apply the same argument to $-S$ and add the bounds to obtain the Hoeffding inequality:If $V_H=0$, every centered $Y_i$ is identically zero and the tail probability is zero; the displayed exponential is interpreted by its zero-variance limit.

For the Bennett inequality, write $v=n{\sigma }^2$, and first assume $M>0$ and $v>0$. For $j\ge 2$, boundedness and centering giveFor $t\ge 0$, expanding the moment-generating function and using the permitted interchange of sum and expected value,Negative odd moments cause no difficulty because the estimate is an upper bound term by term. The independence of the $Y_i$ and the Chernoff bound then giveThe derivative vanishes at $t_{\ast }=M^{-1}\log (1+Mϵ/v)$, and substitution gives $-(v/M^2)\varphi (Mϵ/v)$ with the Bennett rate function $\varphi (u)=(1+u)\log (1+u)-u$. The same absolute bounds apply to $-Y_i$, soIf ${\sigma }^2=0$, all centered $Y_i$ are zero almost surely; if $M=0$, the same is immediate. Handle those cases directly rather than divide by zero. Both derived concentration inequalities also hold with $\ge ϵ$, by the same Markov inequality argument or by letting a smaller positive threshold increase to $ϵ$.

For the binomial distribution, write $X={\sum }_{i=1}^n{Z}_i$ with independent and identically distributed random variables $Z_i$ having the Bernoulli distribution with parameter $p=p_n$. Then $Y_i=Z_i-p$ has range $[-p,1-p]$, width one, and variance $p(1-p)$. For large $n$, $p<1/2$, so $M=1-p$ and $v=np(1-p)$. Set $ϵ=C\sqrt{v}$.

The Hoeffding inequality givesAfter truncation at one, this is asymptotically the trivial probability bound. The Bennett inequality instead givesSince $v\to \infty$ and $\varphi (u)=u^2/2+O(u^3)$ by a Taylor expansion,Thus the Bennett bounds for sparse binomial fluctuations yieldThe Bennett inequality supplies the smaller raw exponential bound for every fixed $C>0$ and all sufficiently large $n$. If $C\le \sqrt{2\log 2}$, clipping probability upper bounds at one makes both eventually trivial; for $C>\sqrt{2\log 2}$, the Bennett inequality retains a nontrivial limiting bound. This distinction avoids claiming a probability improvement after clipping when both raw bounds exceed one.