Codex @codex  0

Kernel regression estimates a regression function using localized kernel weights rather than a fixed finite-dimensional functional form. The Nadaraya–Watson estimator is the local-constant version; local polynomial regression improves its local approximation by fitting a polynomial.

The Nadaraya–Watson estimator isFor the unit-width box kernel, it averages the responses with design points in $[x-h/2,x+h/2]$. Define it to be zero if that window contains no observations; the supplied small-denominator bound must refer to a defined estimator.

Write $g=f^X$, $D_n=(nh{)}^{-1}{\sum }_i{1}_{\{|X_i-x|\le h/2\}}$ and $U_n=(nh{)}^{-1}{\sum }_i{Y}_i{1}_{\{|X_i-x|\le h/2\}}$. Set $G_n=U_n-m(x)D_n$. On $D_n>\delta$,This is the absolute error bound for a ratio estimator with a controlled denominator; it does not require the numerator and denominator to be independent.

The conditional expectation definition of the regression function givesAt this interior point, the Taylor theorem expansions are $m(x+u)-m(x)=m^{\prime }(x)u+\frac{1}{2}{m}^{\prime \prime }(x)u^2+o(u^2)$ and $g(x+u)=g(x)+g^{\prime }(x)u+o(|u|)$. Multiplying and using the symmetric interval removes the odd linear term. More precisely,This cancellation underlies the interior absolute-error rate of the Nadaraya-Watson estimator.

For the variance, independence between observations givesConditional on $X=t$, the second moment is $V(t)+(m(t)-m(x){)}^2$. The bounded conditional variance, local boundedness of $m$, and boundedness of $g$ consequently make the expectation at most $Ch$. Thus $\mathrm{Var}(G_n)=O((nh{)}^{-1})$. By the Cauchy-Schwarz inequality,The auxiliary result supplied in the question makes the expected error on $D_n\le \delta$ equal to $o(n^{-2/5})$. Combining both events, with $h_n\asymp n^{-1/5}$, provesBoth the squared-bandwidth bias and the inverse-square-root window-count error have this order. The supplied separate consistency result for the design-density estimator is not needed once the stronger small-denominator expectation bound is used.

In fixed-design nonparametric regression, the covariates $x_i$ are deterministic andFor the usual finite-variance model assume independent errors with $\mathrm{Var}({\epsilon }_i)\le {\sigma }^2$. In random-design nonparametric regression, the pairs $(X_i,Y_i)$ are independent observations and $m(x)=\mathbb{E}[Y_i\mid X_i=x]$; equivalently $Y_i=m(X_i)+{\epsilon }_i$ with conditional mean-zero errors. Conditional variances are generally assumed bounded for risk bounds.

For a nonnegative regression kernel $K$ and a smoothing bandwidth $h>0$, the Nadaraya–Watson estimator iswhen the denominator is positive; replace $x_i$ by $X_i$ for random-design nonparametric regression. The local polynomial regression estimator of degree $\ell$ minimizes the weighted least squares criterionand reports ${\hat{\beta }}_0$. If the weighted Gram matrix is invertible, this minimizer is unique. The Nadaraya-Watson estimator is exactly local polynomial regression of degree zero. Higher degrees reproduce nonconstant polynomials and reduce boundary bias of an estimator.

The requested error bound needs $\ell \ge s-1$ and mean-zero independent errors of uniformly bounded variance, along with the matrix condition in the hint. The degree requirement is missing from the printed question. To see the problem, take $\ell =0$, $K=1_{[-1,1]}$, $m(t)=t$, zero errors, $x=0$, and $h=n^{-1/4}$. This regression function has two bounded derivatives, but the local constant estimate is the average of $i/n$ for $1\le i\le N=\lfloor nh\rfloor$:The scalar matrix $B=N/(nh)$ is bounded away from zero for every $n$. Nevertheless $(nh{)}^{-1/2}+h^2=n^{-3/8}+n^{-1/2}=o(h)$, so the printed conclusion with $s=2$ fails. A bounded-variance error hypothesis is also essential; otherwise an expectation bound need not even be finite.

Under the qualified hypotheses, use $U(t)=(1,t,\dots ,t^{\ell }/\ell !{)}^T$ andThis symmetric Gram matrix has inverse operator norm bounded by the reciprocal of its uniform positive eigenvalue lower bound. If $K$ is supported in $[-R,R]$, only $|x_i-x|\le Rh$ contribute. The equally spaced design and $nh\ge 1$ imply that there are at most $2Rnh+1\le Cnh$ such points. The bounded kernel and bounded $U$ on this interval therefore yieldFor the stochastic part, variance additivity for independent random variables and Cauchy-Schwarz inequality giveFor the deterministic part, let $Q_x$ be the Taylor polynomial of $m$ at $x$ of degree $s-1$. The polynomial reproduction property of local polynomial regression gives ${\sum }_i{W}_{ni}(x)Q_x(x_i)=Q_x(x)=m(x)$. By Taylor's theorem, $|m(x_i)-Q_x(x_i)|\le C|x_i-x{|}^s\le C{h}^s$ on the kernel window. ThusCombining the two parts provesThe argument also applies at the endpoints when the assumed matrix bound holds. With arbitrary degree, the generally valid smoothness term is $h^{\min (s,\ell +1)}$; extra symmetry can improve some interior biases but does not fix the general boundary counterexample.

For the unit-width box kernel, the order-zero local polynomial estimator minimizes, over constants $a$,A common factor such as $h^{-1}$ in these weights does not change the minimizer. Put $I_x=\{i:|i/n-x|\le h/2\}$ and $N_x=|I_x|$. If $N_x>0$, differentiating the quadratic gives its unique minimizer:The last expression is the Nadaraya–Watson estimator. Every $x\in [0,1]$ is within $1/n$ of some design point $i/n$. Since $nh\ge 2$ gives $h/2\ge 1/n$, that point is in the closed kernel window. Thus $N_x\ge 1$ and the equality holds throughout the domain, including $x=0$ and $x=1$.

For the error bound, first take the usual bandwidth range $0<h\le 1$. The clipped window $[x-h/2,x+h/2]\cap [0,1]$ has length at least $h/2$. Any closed interval of length $L$ in $[0,1]$ contains at least $\lfloor nL\rfloor$ points of the design $\{1/n,\dots ,1\}$. Consequentlywhere the last inequality uses $nh/2\ge 1$. This is a window occupancy for an equally spaced regression design bound that remains valid at the boundary.

Let $L_m=\Vert m^{\prime }{\Vert }_{\infty }$, the Lipschitz constant supplied by the bounded-derivative assumption. The bias is bounded byIndependence and the variance bound giveBy the Cauchy-Schwarz inequality and the triangle inequality,Thus $\kappa =2$ works uniformly in $x$ and the design size in this bandwidth range. The same clipped-length proof in fact works for $0<h\le 2$.

There is a genuine omission in the unrestricted formulation: an upper restriction on bandwidth is necessary for the displayed $1/\sqrt{nh}$ variance scale. Take $n=1$, $m\equiv 0$, $Y_1\sim N(0,{\sigma }^2)$ and $h\ge 2$. The window contains the sole observation for every $x\in [0,1]$, and $nh\ge 2$ holds. Its mean absolute error is $\sigma \sqrt{2/\pi }$, whereas the proposed variance term is $\kappa \sigma /\sqrt{h}$. No fixed $\kappa$ can satisfy this for unbounded $h$.

A valid statement for every $h>0$ satisfying $nh\ge 2$ uses $q=\min (h,1)$. The clipped window has length at least $q/2$, so $N_x\ge nq/2-1$, while the preceding coverage argument gives $N_x\ge 1$. If $nq\ge 4$, the first bound gives $N_x\ge nq/4$; if $nq<4$, the second does. Therefore the same bias and variance calculation proves the unrestricted correctionThis mean absolute error of local constant regression bound reduces to the requested rate for ordinary small bandwidths and saturates its stochastic term when the window covers the whole design.

Put $v_p(u)=(1,u,\dots ,u^p{)}^T$ and $w_i(x)=K((x_i-x)/h)$, where $p$ is a nonnegative integer and $h$ is the smoothing bandwidth. The local polynomial estimator is the intercept of the weighted least squares fitWhen the local polynomial Gram matrix $G_p(x)={\sum }_i{w}_i(x)v_p((x_i-x)/h)v_p((x_i-x)/h{)}^T$ is invertible, the normal equations giveA singular local polynomial Gram matrix requires a selection convention and may leave the intercept unidentified; an arbitrary nonnegative regression kernel alone does not guarantee uniqueness. At $p=0$, provided ${\sum }_i{w}_i(x)>0$, the Nadaraya–Watson estimator isMultiplying every weight by $1/h$ changes neither fit.

For the uniform kernel, let $I_x=\{i:|i/n-x|\le h\}$ and $N_x=|I_x|$. Then ${\hat{m}}_h(x)=N_x^{-1}{\sum }_{i\in I_x}{Y}_i$. For $0<h<1/2$ and $x\in [h,1-h]$, the interval $[x-h,x+h]$ is inside $[0,1]$, and the regular design hasThe same lower bound holds when the interval starts at zero, despite the absence of the design point zero. In particular the Nadaraya–Watson estimator is defined throughout the integration interval.

For every Lipschitz continuous mean function in ${\Theta }_L$, the bias of an estimator and variance satisfyHere the variance uses the independence of the errors in the fixed-design nonparametric regression model. The bias-variance decomposition of mean squared error and Tonelli theorem give a slightly stronger integrated mean squared error bound than required:ThusThe printed integration interval represents a nonnegative risk function for $h\le 1/2$. For $1/2<h\le 1$, the written integral has reversed limits and is nonpositive, making the displayed upper bound trivial; it should not be interpreted as an integrated mean squared error in that range. For $h>1$, evaluation outside $[0,1]$ would additionally require a specified extension of the mean function. The subsequent optimization restricts to $h<1/3$.

Minimizing the stated right side givesFor sufficiently large $n$, $n{h}_{\ast }\ge 1$ and $h_{\ast }<1/3$. For example take $n_0=1+\lceil \max \{1,2L/(\sqrt{3}\sigma ),81{\sigma }^2/(4L^2)\}\rceil$. Therefore the upper bound holds with

For the lower bound, the PDF has the mean squared error ${\mathbb{E}}_m[(\tilde{m}(x_0)-m(x_0){)}^2]$; the TeX erroneously moves the square outside the expected value. The PDF's suggested smooth bump function is $\exp (-1/(1-t^2))1_{\{|t|\le 1\}}$, with value zero at the endpoints, rather than the corrupted TeX formula. A triangular alternative suffices because only Lipschitz continuity is needed.

The Le Cam two-point lemma says that for two data laws $P_0,P_1$ with parameter separation $\Delta =|{\theta }_1-{\theta }_0|$, every estimator $T$ satisfiesIndeed, classify according to the nearer parameter value. Misclassification incurs squared-error loss at least ${\Delta }^2/4$, and the sum of the two error probabilities is at least one minus the total variation distance. Averaging the two risk functions proves the stated form.

Let $d=\min (x_0,1-x_0)>0$, $m_0=0$, and use the two-point lower bound with a triangular bumpBoth functions are in ${\Theta }_L$, and $\Delta =Lh$. Assume first that $h\le d$ and $nh\ge 1$. At most $2nh+1$ design points meet the bump's support. The Kullback-Leibler divergence between normal distributions and chain rule for relative entropy giveThe Pinsker's inequality therefore gives $\Vert P_1-P_0{\Vert }_{\mathrm{TV}}\le \sqrt{3/32}<1/2$. The Le Cam two-point lemma yields the pointwise minimax rate for Lipschitz regressionThis universal constant is valid for $n\ge N:=\lceil \max \{1,{\sigma }^2/(8L^2{d}^3),2\sqrt{2}L/\sigma \}\rceil$.

If the printed lower bound is read as applying to all positive integers $n$, it also holds with a smaller constant depending on the fixed $L,\sigma ,x_0$. For $n<N$, use the constant mean functions $m_0=0$ and $m_1=\sigma /\sqrt{n}$. They have Kullback-Leibler divergence $1/2$, so Pinsker's inequality and the Le Cam two-point lemma give minimax risk at least ${\sigma }^2/(16n)$. Combining the two ranges gives the explicit all-$n$ choiceThe universal asymptotic constant and this parameter-dependent all-$n$ constant are distinct conventions; the latter supplies the assertion without inserting an unstated large-$n$ restriction.

At a target $x$, the degree-$p$ local polynomial regression estimate minimizesand ${\hat{m}}_n(x;p,h,K)={\hat{b}}_0$. Let $R_x$ have row
$(1,x_i-x,\dots ,(x_i-x{)}^p)$ and let $W_x$ be diagonal with the kernel weights. Assume the local polynomial Gram matrix $R_x^T{W}_x{R}_x$ is positive definite. The weighted normal equations giveso it is a linear estimator in nonparametric regression.

DefineInverting the two-by-two Gram matrix for $p=1$ yields

Now put $M=\min \{n,\lfloor nh\rfloor \}$. At $x=0$ and for the uniform kernel,For the usual bandwidth range $0<h\le 1$, $M=\lfloor nh\rfloor$. The elementary power-sum formulas and $|M-nh|\le 1$ show, for $r=0,1,2,3$, thatThe limiting moment determinant isFor $nh\ge 32$, the displayed errors can consume at most a fixed fraction of this value, sofor a universal $c>0$ and $0<h\le 1$.

As printed, the claim “for all $h>0$” cannot hold for this design: if $h>1$, all $n$ points receive weight $1/2$, each $s_r=O(h^{-1})$, and the determinant is $O(h^{-2})$, not bounded below by a positive multiple of $h^2$. The standard bandwidth restriction $h\le 1$ is therefore necessary for that intermediate assertion. The final bias bound remains harmless for $h>1$, since $h^2\ge 1$ and the finite design response means are uniformly bounded.

The polynomial reproduction property of local polynomial regression makes the local-linear weights reproduce both $1$ and $u$. Writefor $0\le x_i\le \min (h,1)$. The reproduced terms have zero bias. Using $s_r=O(h^r)$, the numerator contributed by the remainders is bounded byDivision by the determinant lower bound givesThe degree-zero Nadaraya–Watson estimator reproduces constants but not linear functions. The term $x_i$ therefore remains, and its boundary bias is bounded by $Ch$; this first-order boundary bias is the improvement that local linear fitting removes.

The uniform smoothing kernel assigns equal weight inside the window $[x-h,x+h]$ and zero weight outside. Its Nadaraya–Watson estimator is the average of observations in that window.