Nadaraya–Watson estimator

This local-constant estimator is one particular kernel regression method.

The Nadaraya–Watson estimator is the degree-zero local polynomial estimateAt a support boundary it generally has first-order bias because it reproduces constants but not linear functions.

For the unit-width symmetric box kernel for density estimation, suppose the design density is positive and differentiable at $x$, the regression function is twice differentiable there, and the conditional variance and design density are bounded near $x$. Write $G_n=(nh{)}^{-1}{\sum }_i(Y_i-m(x))1_{\{|X_i-x|\le h/2\}}$. Symmetry cancels its linear Taylor term, giving $\mathbb{E}{G}_n=O(h^2)$; independence gives $\mathrm{Var}{G}_n=O((nh{)}^{-1})$. The absolute error bound for a ratio estimator with a controlled denominator gives the displayed rate if the small-denominator contribution is negligible. The balance $h\asymp n^{-1/5}$ gives $O(n^{-2/5})$. Interior design positivity matters: support boundaries can prevent the linear cancellation.

A local constant estimator reproduces constants but need not reproduce an affine regression function near a support boundary. Let $x_i=i/n$, take $0<h\le 1$ and the regression kernel $K=1_{[-1,1]}$, and suppose $m(x)=x$. At target zero, the noiseless Nadaraya–Watson estimator averages $i/n$ for $1\le i\le k=\lfloor nh\rfloor$. Whenever $k\ge 1$, its exact fitted value isSince $m(0)=0$, this is the bias of an estimator when observation errors have zero expected value. Under $h\to 0$ and $nh\to \infty$, it is asymptotic to $h/2$. The one-sided boundary window has a nonzero first coordinate moment, explaining the first-order bias. An invertible local linear regression fit reproduces $m(x)=x$ exactly at the target and removes this term. If $k=0$, the local constant average is undefined rather than zero, so a nonempty window is a necessary condition.

For fixed $\alpha \in [0,1)$, design $x_i=i/n$, $m\in C^2[0,1]$, a nonzero nonnegative symmetric piecewise $C^1$ regression kernel supported on $[-1,1]$, and $h\to 0$, $n{h}^2\to \infty$, write ${\mu }_{r,\alpha }={\int }_{-\alpha }^1{u}^{r}K(u)du$ and ${\beta }_r={\mu }_{r,\alpha }/{\mu }_{0,\alpha }$. On the rescaled grid of spacing $1/(nh)$, the Riemann sum error for $u^{r}K(u)$ is $O((nh{)}^{-1})$, by bounded variation. The normalized first and second local moments are therefore $h{\beta }_1+O(n^{-1})$ and $h^2{\beta }_2+O(h/n)$. A uniform second-order Taylor expansion gives the displayed bias of an estimator, since $n^{-1}=o(h^2)$. Equivalently it is $h{\beta }_1{m}^{\prime }(0)+h^2(\alpha {\beta }_1+{\beta }_2/2)m^{\prime \prime }(0)+o(h^2)$. The first-order term explains the generic boundary bias of a local constant estimator.

For an equally spaced design with independent errors of variance at most ${\sigma }^2$ and a regression function with Lipschitz constant $L$, the unit-width box kernel estimator averages the $N_x$ observations in the window. Its bias is at most $Lh/2$, and its variance is at most ${\sigma }^2/N_x$. The window occupancy for an equally spaced regression design gives, whenever $nh\ge 2$,In the usual range $h\le 1$, this is $O(\sigma /\sqrt{nh}+Lh)$ uniformly up to the boundary. For arbitrarily large bandwidth, the stochastic error cannot continue to decrease as $1/\sqrt{nh}$: once all observations are included the estimator is simply their mean.

Let $x$ be an interior target for fixed-design nonparametric regression with equally spaced sites, regression function $m$ differentiable at $x$, and independent mean-zero errors of common variance ${\sigma }^2<\infty$. Use a nonnegative symmetric unit-integral compactly supported piecewise smooth regression kernel $K$, with $h\to 0$ and $nh\to \infty$. For the Nadaraya–Watson estimator,The noise variance factor is essential; $R(K)/(nh)$ alone assumes unit variance.

For the bias, expand $m(x_i)=m(x)+m^{\prime }(x)(x_i-x)+o(|x_i-x|)$ uniformly in the shrinking window. Symmetry cancels the limiting first moment and the grid error is $O(1/n)=o(h)$. For the variance, independent errors give ${\sigma }^2{\sum }_i{W}_i(x{)}^2$; the two kernel Riemann sums yield ${\sum }_i{W}_i^2=R(K)/(nh)+o((nh{)}^{-1})$. Hence the squared bias of an estimator is $o(h^2)$. This is a pointwise differentiability argument; it does not supply a uniform remainder over an arbitrary class of functions.