Local polynomial regression

local polynomial regression fits a low-degree polynomial by weighted least squares near each target point.

For equally spaced fixed-design nonparametric regression, independent mean-zero uniformly finite-variance errors, a bounded compactly supported kernel, and a uniformly invertible local Gram matrix, Degree $\ell \ge s-1$ gives the displayed bound for $s$ bounded derivatives. Kernel weights have absolute sum $O(1)$ and squared sum $O((nh{)}^{-1})$. Polynomial reproduction removes a degree-$s-1$ Taylor polynomial. Without the Degree requirement the general bias of an estimator is only $O(h^{\min (s,\ell +1)})$.

local linear regression fits $a+b(X_i-x)$ to responses near each target $x$ by weighted least squares, then returns the fitted intercept. With weights $w_i=K((X_i-x)/h)$, put $S_k={\sum }_i{w}_i(X_i-x{)}^k$ and $T_k={\sum }_i{w}_i(X_i-x{)}^k{Y}_i$. If $S_0{S}_2-S_1^2>0$, then $\hat{f}(x)=(S_2{T}_0-S_1{T}_1)/(S_0{S}_2-S_1^2)$. Its polynomial reproduction property of local polynomial regression reproduces affine functions, reducing boundary bias relative to a local constant fit.

A nonnegative regression kernel supplies weights $K((x_i-x)/h)$ in a local weighted least squares fit. Unlike a probability density function, its integral need not equal one, because rescaling every weight does not change the fitted coefficients.

A smoothing bandwidth $h>0$ determines the scale over which a regression kernel averages data, or a kernel for density estimation spreads each observation. It has the same units as the observations. In kernel density estimation, changing $h$ changes the visible detail while preserving the total integral one. For a Gaussian density kernel, each bump has standard deviation $h$ and its individual contribution has peak height $1/(nh\sqrt{2\pi })$.

If the smoothing bandwidth is too small relative to the spacings in a statistical sample, the kernel density estimator shows narrow spikes tied to individual observations: apparent peaks may be sampling accidents. If it is too large relative to features of the unknown probability density function, it blurs distinct groups together and spreads fitted probability far from the observations. An intermediate smoothing bandwidth can reveal broad structure, but visual smoothness alone cannot establish accuracy.

The first three panels use exactly $(-1.8,-1.4,-1.1,-0.7,0.7,1.1,1.4,1.8)$ with smoothing bandwidths $0.08$, $0.35$ and $1.2$, respectively. Every curve has area one; the two groups in the observed statistical sample become less distinct as $h$ increases. No true probability density function is specified in this illustration, so the middle panel is illustrative, not a certified optimal fit. The fourth panel uses $(0.1,0.4,0.9)$ and $h=0.5$ to illustrate boundary bias of a kernel density estimator.

For a fixed nonnegative kernel for density estimation $K$ with finite squared integral, a probability density function $f$ that is bounded and continuous at an interior point $x$ with $f(x)>0$, and a deterministic sequence $h=h_n\downarrow 0$, the pointwise sampling variance obeysThus decreasing the smoothing bandwidth increases the leading sampling variance while reducing interior smoothing bias of an estimator: this is a bias-variance tradeoff. The conditions $h_n\to 0$ and $n{h}_n\to \infty$ express two goals, increasingly fine resolution and enough observations contributing locally. Practical methods such as plug-in bandwidth selection use the data to choose $h$; pointwise optimal kernel bandwidth depends on local features of the unknown probability density function. The smoothing bandwidth remains a tuning choice in nonparametric statistics, and knowing the support of a probability distribution can require more than a width adjustment near a boundary.

Estimate the unknown curvature in the AMISE minimizer and substitute it into that formula. A consistent positive curvature estimate makes the selected bandwidth ratio-consistent. Pilot smoothing is needed because differentiating an undersmoothed density estimator magnifies variance. A normal-reference pilot can initialize an estimate of a higher derivative functional; the final density-curvature estimate is computed from the data rather than assuming the density is normal.

Twice integrating by parts gives $R(f^{\prime \prime })=\int f{f}^{(4)}$. Estimate this by averaging a fourth-derivative kernel density estimator at the observations. Its expectation contains diagonal term $K^{(4)}(0)/(n{g}^5)$ and leading smoothing bias $-g^2{\mu }_2(K)R(f^{\prime \prime \prime })/2$. Balancing them yields pilot scale $g\asymp R(f^{\prime \prime \prime }{)}^{-1/7}{n}^{-1/7}$. With suitable derivatives and tails, the estimation error is bounded by $O_p(g^2+(n{g}^5{)}^{-1}+n^{-1/2}+n^{-1}{g}^{-9/2})$, giving an $O_p(n^{-2/7})$ relative-bandwidth bound after substitution. More carefully calibrated bias cancellation can improve that bound.

// Target: statistical-inference.bigb

The uniform smoothing kernel assigns equal weight inside the window $[x-h,x+h]$ and zero weight outside. Its Nadaraya–Watson estimator is the average of observations in that window.

The unit-width centred box kernel is $K(u)=1_{[-1/2,1/2]}(u)$. It has integral one and squared $L^2$ norm one. Its rescaling $K_h(u)=h^{-1}K(u/h)$ averages a function on intervals of length $h$. It is obtained from the uniform smoothing kernel with support $[-1,1]$ by rescaling, and is useful in both kernel density estimation and local polynomial regression.

The local polynomial Gram matrix is the weighted matrix of inner products of the local monomial basis. Its invertibility makes the local fit unique.

The effective kernel weights express a local polynomial regression fit as a linear combination of the responses. At target $x$, set $z_i=(x_i-x)/h$, $r_i=(1,z_i,\dots ,z_i^p{)}^T$, and $w_i=K(z_i)$. If the local polynomial Gram matrix $B={\sum }_i{w}_i{r}_i{r}_i^T$ is invertible, thenwhere $e_0=(1,0,\dots ,0{)}^T$. This follows by solving the weighted least squares normal equations and taking the fitted intercept.

The effective kernel weights are not the raw weights $K(z_i)$. They depend on the whole local design and satisfy the polynomial reproduction identities ${\sum }_i{W}_i=1$ and ${\sum }_i{W}_i{z}_i^j=0$ for $1\le j\le p$. Even when $K\ge 0$, individual $W_i$ can be negative. For a local linear regression at $x=0$ through observations at $1,2$, the weights are $2,-1$, giving intercept $2Y_1-Y_2$. The negative weight is what permits exact extrapolation of a straight line; forcing all weights positive would lose that property at this target.

Local polynomial weights of Degree $p$ reproduce every multivariate polynomial of total degree at most $p$: applying the estimator to its design values returns its value at the target point.

If a degree-$p$ local polynomial is fitted in the scaled coordinate $(x_i-x)/h$, its coefficient of that coordinate divided by $h$ estimates the first derivative. Its effective weights differentiate every polynomial of Degree at most $p$ exactly at the target.

This local-constant estimator is one particular kernel regression method.

The Nadaraya–Watson estimator is the degree-zero local polynomial estimateAt a support boundary it generally has first-order bias because it reproduces constants but not linear functions.

For the unit-width symmetric box kernel for density estimation, suppose the design density is positive and differentiable at $x$, the regression function is twice differentiable there, and the conditional variance and design density are bounded near $x$. Write $G_n=(nh{)}^{-1}{\sum }_i(Y_i-m(x))1_{\{|X_i-x|\le h/2\}}$. Symmetry cancels its linear Taylor term, giving $\mathbb{E}{G}_n=O(h^2)$; independence gives $\mathrm{Var}{G}_n=O((nh{)}^{-1})$. The absolute error bound for a ratio estimator with a controlled denominator gives the displayed rate if the small-denominator contribution is negligible. The balance $h\asymp n^{-1/5}$ gives $O(n^{-2/5})$. Interior design positivity matters: support boundaries can prevent the linear cancellation.

A local constant estimator reproduces constants but need not reproduce an affine regression function near a support boundary. Let $x_i=i/n$, take $0<h\le 1$ and the regression kernel $K=1_{[-1,1]}$, and suppose $m(x)=x$. At target zero, the noiseless Nadaraya–Watson estimator averages $i/n$ for $1\le i\le k=\lfloor nh\rfloor$. Whenever $k\ge 1$, its exact fitted value isSince $m(0)=0$, this is the bias of an estimator when observation errors have zero expected value. Under $h\to 0$ and $nh\to \infty$, it is asymptotic to $h/2$. The one-sided boundary window has a nonzero first coordinate moment, explaining the first-order bias. An invertible local linear regression fit reproduces $m(x)=x$ exactly at the target and removes this term. If $k=0$, the local constant average is undefined rather than zero, so a nonempty window is a necessary condition.

For fixed $\alpha \in [0,1)$, design $x_i=i/n$, $m\in C^2[0,1]$, a nonzero nonnegative symmetric piecewise $C^1$ regression kernel supported on $[-1,1]$, and $h\to 0$, $n{h}^2\to \infty$, write ${\mu }_{r,\alpha }={\int }_{-\alpha }^1{u}^{r}K(u)du$ and ${\beta }_r={\mu }_{r,\alpha }/{\mu }_{0,\alpha }$. On the rescaled grid of spacing $1/(nh)$, the Riemann sum error for $u^{r}K(u)$ is $O((nh{)}^{-1})$, by bounded variation. The normalized first and second local moments are therefore $h{\beta }_1+O(n^{-1})$ and $h^2{\beta }_2+O(h/n)$. A uniform second-order Taylor expansion gives the displayed bias of an estimator, since $n^{-1}=o(h^2)$. Equivalently it is $h{\beta }_1{m}^{\prime }(0)+h^2(\alpha {\beta }_1+{\beta }_2/2)m^{\prime \prime }(0)+o(h^2)$. The first-order term explains the generic boundary bias of a local constant estimator.

For an equally spaced design with independent errors of variance at most ${\sigma }^2$ and a regression function with Lipschitz constant $L$, the unit-width box kernel estimator averages the $N_x$ observations in the window. Its bias is at most $Lh/2$, and its variance is at most ${\sigma }^2/N_x$. The window occupancy for an equally spaced regression design gives, whenever $nh\ge 2$,In the usual range $h\le 1$, this is $O(\sigma /\sqrt{nh}+Lh)$ uniformly up to the boundary. For arbitrarily large bandwidth, the stochastic error cannot continue to decrease as $1/\sqrt{nh}$: once all observations are included the estimator is simply their mean.

Let $x$ be an interior target for fixed-design nonparametric regression with equally spaced sites, regression function $m$ differentiable at $x$, and independent mean-zero errors of common variance ${\sigma }^2<\infty$. Use a nonnegative symmetric unit-integral compactly supported piecewise smooth regression kernel $K$, with $h\to 0$ and $nh\to \infty$. For the Nadaraya–Watson estimator,The noise variance factor is essential; $R(K)/(nh)$ alone assumes unit variance.

For the bias, expand $m(x_i)=m(x)+m^{\prime }(x)(x_i-x)+o(|x_i-x|)$ uniformly in the shrinking window. Symmetry cancels the limiting first moment and the grid error is $O(1/n)=o(h)$. For the variance, independent errors give ${\sigma }^2{\sum }_i{W}_i(x{)}^2$; the two kernel Riemann sums yield ${\sum }_i{W}_i^2=R(K)/(nh)+o((nh{)}^{-1})$. Hence the squared bias of an estimator is $o(h^2)$. This is a pointwise differentiability argument; it does not supply a uniform remainder over an arbitrary class of functions.