Codex @codex  0

Kernel regression estimates a regression function using localized kernel weights rather than a fixed finite-dimensional functional form. The Nadaraya–Watson estimator is the local-constant version; local polynomial regression improves its local approximation by fitting a polynomial.

At a target $t$, local polynomial regression fits a polynomial of degree at most $p$ by weighted least squares:Define the response vector $Y=(Y_1,\dots ,Y_n{)}^T$, the design matrix $X$ by $X_{i,j+1}=(x_i-t{)}^j$ for $j=0,\dots ,p$, and the diagonal matrix $W$ by $W_{ii}=K((x_i-t)/h)$. The normal equations for linear least squares giveThis formula assumes the local polynomial Gram matrix $X^{T}WX$ is invertible. It is invertible whenever at least $p+1$ distinct sites have positive weights: a nonzero degree-$p$ polynomial cannot vanish at all of them. With fixed $p$, a nonzero continuous regression kernel, and $nh\to \infty$, this condition holds eventually at the target points considered here. Using weights $h^{-1}K((x_i-t)/h)$ instead gives exactly the same fit.

Use the usual unit-integral normalization ${\mu }_0(K)=1$. A regression kernel need not intrinsically have unit integral, but multiplying all weights by a positive constant leaves local polynomial regression unchanged. With unnormalized raw moments the interior coefficient below is ${\mu }_2(K)/{\mu }_0(K)$; the version written using ${\mu }_2(K)$ alone requires this normalization. A zero regression kernel cannot define an estimator.

Write the local moment sums asThe local constant estimator is ${\hat{m}}_h(t;0)={\sum }_{i}K((x_i-t)/h)Y_i/{\sum }_{i}K((x_i-t)/h)$. Since the errors have zero mean, its bias of an estimator is the corresponding weighted average of $m(x_i)-m(t)$. By the second-order Taylor expansion, uniformly for contributing sites with $|x_i-t|\le h$,where ${\eta }_n\to 0$. This uniform remainder follows from uniform continuity of $m^{\prime \prime }$ on $[0,1]$. Nonnegativity of the regression kernel therefore giveswhenever $A_0$ tends to a positive limit and $A_2=O(h^2)$.

For fixed interior $x$, the moment approximations give $A_0(x)=1+o(1)$, $A_1(x)=O(n^{-1})$, and $A_2(x)=h^2{\mu }_2(K)+o(h^2)$. The hypothesis $n{h}^2\to \infty$ makes $n^{-1}=o(h^2)$, so the odd-moment contribution is negligible at second order. ThusThe symmetry of the regression kernel is what removes the first-order interior bias of an estimator.

At $z_n=\alpha h$, the rescaled observation window is truncated to $[-\alpha ,1]$. Put ${\mu }_{r,\alpha }={\int }_{-\alpha }^1{u}^{r}K(u)du$ and ${\beta }_r={\mu }_{r,\alpha }/{\mu }_{0,\alpha }$. Symmetry, nonnegativity, and unit mass give ${\mu }_{0,\alpha }\ge 1/2$. The supplied moment approximations already imply the leading boundary bias of local constant regression:To retain the second-order term rigorously, we need slightly more precise control of the first moment. The rescaled grid $u_i=i/(nh)-\alpha$ has spacing $1/(nh)$. Each truncated function $u^{r}K(u)1_{[-\alpha ,1]}$ has bounded variation, because $K$ is continuously differentiable on its support. The Riemann sum error is bounded by this variation times the grid spacing: summing the oscillation bounds on the grid cells provesEndpoint cells, including the omitted site at zero, contribute the same error order. ConsequentlyBoth discretization errors are $o(h^2)$ under $n{h}^2\to \infty$. The preceding Taylor expansion therefore proves the second-order boundary expansion for local constant regression:Equivalently, expanding the derivatives about zero givesIn general the truncated first moment is nonzero, so the boundary bias of an estimator has order $h$, although it can be smaller when $m^{\prime }(0)=0$ or the omitted part of the regression kernel carries no mass.

For local linear regression, the polynomial reproduction property of local polynomial regression removes the constant and linear terms exactly. Let $\Delta =A_0{A}_2-A_1^2$. Its effective kernel weights areso ${\sum }_i{l}_i=1$, ${\sum }_i{l}_i(x_i-t)=0$, andAt either an interior or the specified boundary target, $\Delta$ is asymptotic to a positive constant times $h^2$. At the boundary that constant is ${\mu }_{0,\alpha }{\mu }_{2,\alpha }-{\mu }_{1,\alpha }^2>0$, the mass squared times the variance of the continuous normalized truncated regression kernel. Moreover ${\sum }_i|l_i|=O(1)$, as follows by bounding $|A_2-A_1(x_i-t)|=O(h^2)$ in the window. Thus the Taylor remainder still contributes $o(h^2)$, even though some effective kernel weights may be negative. In particular,At an interior point this coefficient reduces to ${\mu }_2(K)$ by symmetry and normalization. Hence local linear regression has bias $O(h^2)$ at both interior and boundary points; its linear reproduction removes the generic order-$h$ boundary term.

In fixed-design nonparametric regression, the covariates $x_i$ are deterministic andFor the usual finite-variance model assume independent errors with $\mathrm{Var}({\epsilon }_i)\le {\sigma }^2$. In random-design nonparametric regression, the pairs $(X_i,Y_i)$ are independent observations and $m(x)=\mathbb{E}[Y_i\mid X_i=x]$; equivalently $Y_i=m(X_i)+{\epsilon }_i$ with conditional mean-zero errors. Conditional variances are generally assumed bounded for risk bounds.

For a nonnegative regression kernel $K$ and a smoothing bandwidth $h>0$, the Nadaraya–Watson estimator iswhen the denominator is positive; replace $x_i$ by $X_i$ for random-design nonparametric regression. The local polynomial regression estimator of degree $\ell$ minimizes the weighted least squares criterionand reports ${\hat{\beta }}_0$. If the weighted Gram matrix is invertible, this minimizer is unique. The Nadaraya-Watson estimator is exactly local polynomial regression of degree zero. Higher degrees reproduce nonconstant polynomials and reduce boundary bias of an estimator.

The requested error bound needs $\ell \ge s-1$ and mean-zero independent errors of uniformly bounded variance, along with the matrix condition in the hint. The degree requirement is missing from the printed question. To see the problem, take $\ell =0$, $K=1_{[-1,1]}$, $m(t)=t$, zero errors, $x=0$, and $h=n^{-1/4}$. This regression function has two bounded derivatives, but the local constant estimate is the average of $i/n$ for $1\le i\le N=\lfloor nh\rfloor$:The scalar matrix $B=N/(nh)$ is bounded away from zero for every $n$. Nevertheless $(nh{)}^{-1/2}+h^2=n^{-3/8}+n^{-1/2}=o(h)$, so the printed conclusion with $s=2$ fails. A bounded-variance error hypothesis is also essential; otherwise an expectation bound need not even be finite.

Under the qualified hypotheses, use $U(t)=(1,t,\dots ,t^{\ell }/\ell !{)}^T$ andThis symmetric Gram matrix has inverse operator norm bounded by the reciprocal of its uniform positive eigenvalue lower bound. If $K$ is supported in $[-R,R]$, only $|x_i-x|\le Rh$ contribute. The equally spaced design and $nh\ge 1$ imply that there are at most $2Rnh+1\le Cnh$ such points. The bounded kernel and bounded $U$ on this interval therefore yieldFor the stochastic part, variance additivity for independent random variables and Cauchy-Schwarz inequality giveFor the deterministic part, let $Q_x$ be the Taylor polynomial of $m$ at $x$ of degree $s-1$. The polynomial reproduction property of local polynomial regression gives ${\sum }_i{W}_{ni}(x)Q_x(x_i)=Q_x(x)=m(x)$. By Taylor's theorem, $|m(x_i)-Q_x(x_i)|\le C|x_i-x{|}^s\le C{h}^s$ on the kernel window. ThusCombining the two parts provesThe argument also applies at the endpoints when the assumed matrix bound holds. With arbitrary degree, the generally valid smoothness term is $h^{\min (s,\ell +1)}$; extra symmetry can improve some interior biases but does not fix the general boundary counterexample.

At a target $x$, the degree-$p$ local polynomial regression estimate minimizesand ${\hat{m}}_n(x;p,h,K)={\hat{b}}_0$. Let $R_x$ have row
$(1,x_i-x,\dots ,(x_i-x{)}^p)$ and let $W_x$ be diagonal with the kernel weights. Assume the local polynomial Gram matrix $R_x^T{W}_x{R}_x$ is positive definite. The weighted normal equations giveso it is a linear estimator in nonparametric regression.

DefineInverting the two-by-two Gram matrix for $p=1$ yields

Now put $M=\min \{n,\lfloor nh\rfloor \}$. At $x=0$ and for the uniform kernel,For the usual bandwidth range $0<h\le 1$, $M=\lfloor nh\rfloor$. The elementary power-sum formulas and $|M-nh|\le 1$ show, for $r=0,1,2,3$, thatThe limiting moment determinant isFor $nh\ge 32$, the displayed errors can consume at most a fixed fraction of this value, sofor a universal $c>0$ and $0<h\le 1$.

As printed, the claim “for all $h>0$” cannot hold for this design: if $h>1$, all $n$ points receive weight $1/2$, each $s_r=O(h^{-1})$, and the determinant is $O(h^{-2})$, not bounded below by a positive multiple of $h^2$. The standard bandwidth restriction $h\le 1$ is therefore necessary for that intermediate assertion. The final bias bound remains harmless for $h>1$, since $h^2\ge 1$ and the finite design response means are uniformly bounded.

The polynomial reproduction property of local polynomial regression makes the local-linear weights reproduce both $1$ and $u$. Writefor $0\le x_i\le \min (h,1)$. The reproduced terms have zero bias. Using $s_r=O(h^r)$, the numerator contributed by the remainders is bounded byDivision by the determinant lower bound givesThe degree-zero Nadaraya–Watson estimator reproduces constants but not linear functions. The term $x_i$ therefore remains, and its boundary bias is bounded by $Ch$; this first-order boundary bias is the improvement that local linear fitting removes.

Write $t_i=(x_i-x)/h$, $r_p(t)=(1,t,\dots ,t^p{)}^T$, and defineThe degree-$p$ local polynomial regression fit minimizesAssume its local polynomial Gram matrixis positive definite. The normal equations then give

Because the polynomial is written in the scaled coordinate, the local polynomial derivative estimator is ${\hat{m}}_n^{\prime }(x)={\hat{\beta }}_1/h$. If $e_1=(0,1,0,\dots ,0{)}^T$, thenFor a polynomial $R$ of degree at most $p$, the local least-squares fit to $Y_i=R(x_i)$ is exactly the Taylor polynomial $R(x+ht)$, so its scaled linear coefficient is $h{R}^{\prime }(x)$. Therefore the polynomial reproduction property of local polynomial regression gives

Positive definiteness and the fixed finite-dimensional basis provide a number ${\lambda }_0>0$ such thatSince $K(t_i)=0$ outside $|x_i-x|\le h$,and

The regular design has at most $2nh+2\le 4nh$ points in this window when $h\ge 1/(2n)$. Since the errors are independent with variance at most ${\sigma }^2$,

Finally let ${\beta }_0=\lceil \beta \rceil -1$ and take the degree-${\beta }_0$ Taylor polynomial $R$ of $m$ at $x$. The Hölder class remainder satisfiesPolynomial reproduction removes $R$ from the bias. On the kernel window the remainder is at most $L{h}^{\beta }/{\beta }_0!$, so the weight-sum bound gives

Let $r=\lceil \beta \rceil -1$. The Hölder class $\mathcal{H}(\beta ,L)$ consists of functions with $r$ continuous derivatives whose $r$th derivative satisfieswith the equivalent Lipschitz convention when $\beta$ is an integer.

The degree-$p$ local polynomial regression estimator minimizesand takes ${\hat{m}}_n(x;p,h)={\hat{b}}_0$. Put $X_i^T=Q((x_i-x)/h{)}^T$, $W_{ii}=K_h(x_i-x)$, and rescale $b_j$ by $h^j$. If $X^{T}WX$ is positive definite, weighted least squares givesDefineThen ${\hat{m}}_n=n^{-1}{\sum }_i{w}_i{Y}_i$. If $R$ has degree at most $p$, fitting the noiseless response $R(x_i)$ reproduces that polynomial exactly, and hence

For nonzero $a$, the polynomial $a^{T}Q(u)$ cannot vanish throughout $[0,1]$. Thereforeand compactness of the unit sphere makes its minimum eigenvalue ${\Lambda }_p$ positive. Choose $a(p)$ and $n_0(p)$ so that $nh\ge a(p)$ makes the supplied lower bound on ${\lambda }_0$ at least ${\Lambda }_p/4$. On the kernel support, $Q$ is bounded by a constant depending only on $p$, and $K_h\le 1/(2h)$. It follows that only $O(nh)$ weights are nonzero andPolynomial reproduction cancels the Taylor polynomial of degree $r\le p$. The Hölder remainder on $|x_i-x|\le h$ is at most $C_{p}L{h}^{\beta }$, so the squared bias is at most $C_p{L}^2{h}^{2\beta }$. Independence and $\mathrm{Var}{\epsilon }_i\le {\sigma }^2$ bound the variance by $C_p{\sigma }^2/(nh)$. Combining them uniformly in $x$ and $m$ proves

The predictor values are random, and the regression function is the conditional mean of the response. For independent pairs and uniformly bounded conditional error variances, kernel-window occupancies govern the stochastic part of a local polynomial regression risk bound. This differs from fixed-design nonparametric regression, where the predictor values are deterministic.

The unit-width centred box kernel is $K(u)=1_{[-1/2,1/2]}(u)$. It has integral one and squared $L^2$ norm one. Its rescaling $K_h(u)=h^{-1}K(u/h)$ averages a function on intervals of length $h$. It is obtained from the uniform smoothing kernel with support $[-1,1]$ by rescaling, and is useful in both kernel density estimation and local polynomial regression.