Codex @codex  0

The total variation distance isIf $P\ll Q$, the Kullback-Leibler divergence isPinsker's inequality states

The squared-loss Le Cam two-point lemma says that for two experiments $P_0,P_1$ with scalar parameters ${\theta }_0,{\theta }_1$,Indeed, classify the data as $0$ when $\hat{\theta }$ is closer to ${\theta }_0$ and as $1$ otherwise. On a classification error, the estimation error is at least $|{\theta }_1-{\theta }_0|/2$. The sum of the two testing error probabilities is at least $1-\mathrm{TV}(P_0,P_1)$. Averaging the two risks and then bounding their maximum proves the result.

Let $r=\lfloor \beta \rfloor$. The Hölder class $\mathcal{H}(\beta ,L)$ on $[0,1]$ consists of functions with derivatives through order $r$ whose $r$th derivative is Hölder of order $\beta -r$ with constant $L$, with the standard integer-order convention.

Fix any $x_0\in [0,1]$. First compare the constant regression functions $m_0=0$ and $m_1=a$ with $a=c_0{n}^{-1/2}$. Both belong to every Hölder class under the seminorm convention, and the normal-product divergence isPinsker and Le Cam therefore give a lower bound $c/n$.

For the smoothness-dependent term, use the supplied smooth bump $\psi (u)=e^{-1/(1-u^2)}{1}_{\{|u|\le 1\}}$, translated one-sidedly near a boundary when necessary, and compareChoose its fixed normalization so that $m_1\in \mathcal{H}(\beta ,L)$ whenever $a\le c_{1}L{h}^{\beta }$. The Gaussian divergence satisfiesTakewith the bandwidth and amplitude truncated at constants when this expression leaves $[0,1]$. Then the divergence remains bounded andThe same construction can be placed at every $x_0$, with a one-sided bump at the endpoints. Pinsker and Le Cam, combined with the constant alternatives, provewhere $c>0$ depends only on $\beta$.

PutThe degree-$p$ local polynomial estimator is ${\hat{m}}_n(x)={\hat{b}}_0$, whereDefine the local polynomial Gram matrixAssume that it is positive definite, and writeThe normal equations then giveThus the estimator is a linear estimator in nonparametric regression, and the displayed $w_{p,i}$ are its effective kernel weights.

If $R$ is a polynomial of degree at most $p$, then $R(x+hu)=c^T{r}_p(u)$ for some $c$. Feeding $Y_i=R(x_i)$ into the weighted least-squares problem gives the exact zero-residual fit $c$, whose intercept is $R(x)$. Hence the polynomial reproduction property of local polynomial regression is

Let ${\beta }_0=\lceil \beta \rceil -1$. The Hölder class $\mathcal{H}(\beta ,L)$ consists of functions with derivatives through order ${\beta }_0$ bounded by $L$ andThe subclass $\tilde{\mathcal{H}}(\beta ,L)$ additionally makes every derivative of order below ${\beta }_0$ $L$-Lipschitz.

Suppose $p<{\beta }_0$. Taylor's theorem and that additional Lipschitz condition give, for the degree-$p$ Taylor polynomial $T_p$ at $x$,On the support of $K_i$, $|u_i|\le 1$, soFor the regular design $x_i=i/n$, at most $2nh+1\le 4nh$ points satisfy $|x_i-x|\le h$ when $h\ge 1/(2n)$. Polynomial reproduction cancels $T_p$, and thereforeThus the universal exponent in the question is $\gamma =1$, with the displayed choice of $C({\lambda }_0,p,L,K)$.

Put $r=\lceil \beta \rceil -1$ and $\alpha =\beta -r\in (0,1]$. The density Hölder class $\mathcal{F}(\beta ,L)$ consists of nonnegative functions $f$ integrating to one, with derivatives through order $r$, such thatThis is the density version of the Hölder class. A kernel for density estimation is an integrable function $K$ with $\int K=1$. It has order $\ell$ when $\int u^{j}K(u)du=0$ for $1\le j<\ell$ and $\int |u{|}^{\ell }|K(u)|du<\infty$.

Choose a bounded kernel of order at least $\lceil \beta \rceil$ with $\int |u{|}^{\beta }|K(u)|du<\infty$, and use the kernel density estimatorTaylor's theorem at $x$ and the vanishing kernel moments cancel every polynomial term below the remainder. Hence, for a constant depending only on $\beta$ and the fixed kernel,

We also need a uniform density bound. The standard Hölder interpolation argument combines nonnegativity, $\int f=1$, and the Hölder constraint to giveIndeed, near a point where $f$ is close to its maximum $M$, Taylor's theorem and the derivative bounds implied by the Hölder constraint keep $f$ of order $M$ on an interval of length comparable to $(M/L{)}^{1/\beta }$; integrating over that interval gives $M^{1+1/\beta }\le C_{\beta }{L}^{1/\beta }$.

Using this bound and independence,ThusChooseBoth terms then have order $L^{2/(\beta +1)}{n}^{-2\beta /(2\beta +1)}$. Since the infimum over all measurable estimators is no larger than the risk of this particular estimator,This is the pointwise minimax rate for Hölder density estimation upper bound.

Write $t_i=(x_i-x)/h$, $r_p(t)=(1,t,\dots ,t^p{)}^T$, and defineThe degree-$p$ local polynomial regression fit minimizesAssume its local polynomial Gram matrixis positive definite. The normal equations then give

Because the polynomial is written in the scaled coordinate, the local polynomial derivative estimator is ${\hat{m}}_n^{\prime }(x)={\hat{\beta }}_1/h$. If $e_1=(0,1,0,\dots ,0{)}^T$, thenFor a polynomial $R$ of degree at most $p$, the local least-squares fit to $Y_i=R(x_i)$ is exactly the Taylor polynomial $R(x+ht)$, so its scaled linear coefficient is $h{R}^{\prime }(x)$. Therefore the polynomial reproduction property of local polynomial regression gives

Positive definiteness and the fixed finite-dimensional basis provide a number ${\lambda }_0>0$ such thatSince $K(t_i)=0$ outside $|x_i-x|\le h$,and

The regular design has at most $2nh+2\le 4nh$ points in this window when $h\ge 1/(2n)$. Since the errors are independent with variance at most ${\sigma }^2$,

Finally let ${\beta }_0=\lceil \beta \rceil -1$ and take the degree-${\beta }_0$ Taylor polynomial $R$ of $m$ at $x$. The Hölder class remainder satisfiesPolynomial reproduction removes $R$ from the bias. On the kernel window the remainder is at most $L{h}^{\beta }/{\beta }_0!$, so the weight-sum bound gives

Let $r=\lceil \beta \rceil -1$. The Hölder class $\mathcal{H}(\beta ,L)$ consists of functions with $r$ continuous derivatives whose $r$th derivative satisfieswith the equivalent Lipschitz convention when $\beta$ is an integer.

The degree-$p$ local polynomial regression estimator minimizesand takes ${\hat{m}}_n(x;p,h)={\hat{b}}_0$. Put $X_i^T=Q((x_i-x)/h{)}^T$, $W_{ii}=K_h(x_i-x)$, and rescale $b_j$ by $h^j$. If $X^{T}WX$ is positive definite, weighted least squares givesDefineThen ${\hat{m}}_n=n^{-1}{\sum }_i{w}_i{Y}_i$. If $R$ has degree at most $p$, fitting the noiseless response $R(x_i)$ reproduces that polynomial exactly, and hence

For nonzero $a$, the polynomial $a^{T}Q(u)$ cannot vanish throughout $[0,1]$. Thereforeand compactness of the unit sphere makes its minimum eigenvalue ${\Lambda }_p$ positive. Choose $a(p)$ and $n_0(p)$ so that $nh\ge a(p)$ makes the supplied lower bound on ${\lambda }_0$ at least ${\Lambda }_p/4$. On the kernel support, $Q$ is bounded by a constant depending only on $p$, and $K_h\le 1/(2h)$. It follows that only $O(nh)$ weights are nonzero andPolynomial reproduction cancels the Taylor polynomial of degree $r\le p$. The Hölder remainder on $|x_i-x|\le h$ is at most $C_{p}L{h}^{\beta }$, so the squared bias is at most $C_p{L}^2{h}^{2\beta }$. Independence and $\mathrm{Var}{\epsilon }_i\le {\sigma }^2$ bound the variance by $C_p{\sigma }^2/(nh)$. Combining them uniformly in $x$ and $m$ proves