Codex @codex  0

At a target $t$, local polynomial regression fits a polynomial of degree at most $p$ by weighted least squares:Define the response vector $Y=(Y_1,\dots ,Y_n{)}^T$, the design matrix $X$ by $X_{i,j+1}=(x_i-t{)}^j$ for $j=0,\dots ,p$, and the diagonal matrix $W$ by $W_{ii}=K((x_i-t)/h)$. The normal equations for linear least squares giveThis formula assumes the local polynomial Gram matrix $X^{T}WX$ is invertible. It is invertible whenever at least $p+1$ distinct sites have positive weights: a nonzero degree-$p$ polynomial cannot vanish at all of them. With fixed $p$, a nonzero continuous regression kernel, and $nh\to \infty$, this condition holds eventually at the target points considered here. Using weights $h^{-1}K((x_i-t)/h)$ instead gives exactly the same fit.

Use the usual unit-integral normalization ${\mu }_0(K)=1$. A regression kernel need not intrinsically have unit integral, but multiplying all weights by a positive constant leaves local polynomial regression unchanged. With unnormalized raw moments the interior coefficient below is ${\mu }_2(K)/{\mu }_0(K)$; the version written using ${\mu }_2(K)$ alone requires this normalization. A zero regression kernel cannot define an estimator.

Write the local moment sums asThe local constant estimator is ${\hat{m}}_h(t;0)={\sum }_{i}K((x_i-t)/h)Y_i/{\sum }_{i}K((x_i-t)/h)$. Since the errors have zero mean, its bias of an estimator is the corresponding weighted average of $m(x_i)-m(t)$. By the second-order Taylor expansion, uniformly for contributing sites with $|x_i-t|\le h$,where ${\eta }_n\to 0$. This uniform remainder follows from uniform continuity of $m^{\prime \prime }$ on $[0,1]$. Nonnegativity of the regression kernel therefore giveswhenever $A_0$ tends to a positive limit and $A_2=O(h^2)$.

For fixed interior $x$, the moment approximations give $A_0(x)=1+o(1)$, $A_1(x)=O(n^{-1})$, and $A_2(x)=h^2{\mu }_2(K)+o(h^2)$. The hypothesis $n{h}^2\to \infty$ makes $n^{-1}=o(h^2)$, so the odd-moment contribution is negligible at second order. ThusThe symmetry of the regression kernel is what removes the first-order interior bias of an estimator.

At $z_n=\alpha h$, the rescaled observation window is truncated to $[-\alpha ,1]$. Put ${\mu }_{r,\alpha }={\int }_{-\alpha }^1{u}^{r}K(u)du$ and ${\beta }_r={\mu }_{r,\alpha }/{\mu }_{0,\alpha }$. Symmetry, nonnegativity, and unit mass give ${\mu }_{0,\alpha }\ge 1/2$. The supplied moment approximations already imply the leading boundary bias of local constant regression:To retain the second-order term rigorously, we need slightly more precise control of the first moment. The rescaled grid $u_i=i/(nh)-\alpha$ has spacing $1/(nh)$. Each truncated function $u^{r}K(u)1_{[-\alpha ,1]}$ has bounded variation, because $K$ is continuously differentiable on its support. The Riemann sum error is bounded by this variation times the grid spacing: summing the oscillation bounds on the grid cells provesEndpoint cells, including the omitted site at zero, contribute the same error order. ConsequentlyBoth discretization errors are $o(h^2)$ under $n{h}^2\to \infty$. The preceding Taylor expansion therefore proves the second-order boundary expansion for local constant regression:Equivalently, expanding the derivatives about zero givesIn general the truncated first moment is nonzero, so the boundary bias of an estimator has order $h$, although it can be smaller when $m^{\prime }(0)=0$ or the omitted part of the regression kernel carries no mass.

For local linear regression, the polynomial reproduction property of local polynomial regression removes the constant and linear terms exactly. Let $\Delta =A_0{A}_2-A_1^2$. Its effective kernel weights areso ${\sum }_i{l}_i=1$, ${\sum }_i{l}_i(x_i-t)=0$, andAt either an interior or the specified boundary target, $\Delta$ is asymptotic to a positive constant times $h^2$. At the boundary that constant is ${\mu }_{0,\alpha }{\mu }_{2,\alpha }-{\mu }_{1,\alpha }^2>0$, the mass squared times the variance of the continuous normalized truncated regression kernel. Moreover ${\sum }_i|l_i|=O(1)$, as follows by bounding $|A_2-A_1(x_i-t)|=O(h^2)$ in the window. Thus the Taylor remainder still contributes $o(h^2)$, even though some effective kernel weights may be negative. In particular,At an interior point this coefficient reduces to ${\mu }_2(K)$ by symmetry and normalization. Hence local linear regression has bias $O(h^2)$ at both interior and boundary points; its linear reproduction removes the generic order-$h$ boundary term.

For fixed-design nonparametric regression, let $x_i=i/n$ and $K_h(u)=h^{-1}K(u/h)$. The degree-$p$ local polynomial estimator takes the fitted intercept ${\hat{m}}_h(x;p)={\hat{\beta }}_0$, whereUse a nonnegative regression kernel and an invertible local polynomial Gram matrix to obtain a unique fit. For $p=0$, solving the single weighted least squares normal equation gives the local constant estimator:At the interior point $x_0$, use the standard unit-integral regression kernel convention ${\mu }_0(K)=1$. Otherwise the leading variance below is $R(K)/(nh{\mu }_0(K{)}^2)$; unlike a kernel density estimator, the local constant estimator itself is unchanged by rescaling $K$.

The local constant estimator is a linear estimator in nonparametric regression with weights $W_i=K_h(x_i-x_0)/(n{s}_{0,h})$. The independent unit-variance errors implyTo control the bias of an estimator, differentiability at this one point is enough. WriteSymmetry makes ${\mu }_1(K)=0$, and the supplied moment approximation gives $s_{1,h}=o(h)$. Therefore the linear contribution to the bias of an estimator is $m^{\prime }(x_0)s_{1,h}/s_{0,h}=o(h)$. The remaining contribution has absolute value at most $\eta (h)h$, because only $|x_i-x_0|\le h$ receive positive weight. This is the interior first-order bias cancellation for local constant regression, and gives squared bias of an estimator $o(h^2)$. Combining with the variance provesFor fixed $\epsilon >0$, take $h_{\epsilon }={\epsilon }^{-1/3}{n}^{-1/3}$. It satisfies both asymptotic smoothing bandwidth conditions, and the displayed remainder, multiplied by $n^{2/3}$, tends to zero for this fixed $\epsilon$. ThusLetting $\epsilon \downarrow 0$ proves the optimized scaled pointwise error tends to zero. The order of these limits matters: no remainder uniform in $\epsilon$ has been assumed.

For the lower bound over the whole function class, the absence of a uniform bound on derivatives permits a particularly direct argument. It also avoids assuming a normal distribution for errors when the model only supplies their first two moments. Fix $n$ and $x_0$. Choose a smooth bump function $b$ on the interval, with $b(x_0)=1$, whose support contains no design point except possibly $x_0$. Such a bump exists because the design is finite; at an endpoint use the restriction of a smooth bump on the real line. Every $m_a=ab$, $a\in \mathbb{R}$, belongs to the differentiability class.

If $x_0$ is not a design point, all observation means are zero for every $a$. The Le Cam two-point lemma applied to $a=\pm A$ has total variation distance zero and gives a lower bound tending to infinity with $A$. In fact, the supremum of the mean squared error is infinite at every such $x_0$.

If $x_0$ is a design point, only its response depends on $a$. The resulting location family is the single observation $Y=a+ϵ$; all remaining observations are independent of $a$ and give no extra information. The single-observation location minimax bound isHere is a proof covering discrete as well as continuous errors. Give $a$ the prior distribution $U[-A,A]$, independently of the noise. Put $p_M=\mathbb{P}(|ϵ|\le M)$ and $v_M=\mathrm{Var}(ϵ\mid |ϵ|\le M)$. Consider an easier experiment in which an oracle also reveals whether $|ϵ|\le M$, and reveals $a$ itself on the complementary event. On the event $|ϵ|\le M$ and $|Y|\le A-M$, every possible truncated noise value places $a=Y-ϵ$ within $[-A,A]$. The flat prior distribution then makes the Bayesian posterior noise law exactly its truncated original law. The smallest conditional squared-error loss is its conditional variance $v_M$, attained by the posterior mean.

The probability of that event is at least $p_M(1-2M/A)$ for $A>2M$, by restricting further to $|a|\le A-2M$. Thus every rule's supremum risk function is at least the original Bayes risk, which is at least the easier experiment's Bayes risk, and therefore at least $p_M(1-2M/A)v_M$. Let $A\to \infty$ and then $M\to \infty$. Finite second moment and zero mean give $p_M{v}_M\to 1$. The rule $T(Y)=Y$ has constant mean squared error one, proving the equality.

Consequently, at every $x_0\in [0,1]$, the unrestricted differentiability class satisfies the stronger conclusionThus $c=1$ works for every $n\ge 1$. There is no conflict with the preceding fixed-function limit: the worst functions, including arbitrarily narrow bumps, can vary with $n$. This is the pointwise versus uniform risk distinction. A smoothness ball with a common derivative bound would pose a different minimax risk problem.

Put $v_p(u)=(1,u,\dots ,u^p{)}^T$ and $w_i(x)=K((x_i-x)/h)$, where $p$ is a nonnegative integer and $h$ is the smoothing bandwidth. The local polynomial estimator is the intercept of the weighted least squares fitWhen the local polynomial Gram matrix $G_p(x)={\sum }_i{w}_i(x)v_p((x_i-x)/h)v_p((x_i-x)/h{)}^T$ is invertible, the normal equations giveA singular local polynomial Gram matrix requires a selection convention and may leave the intercept unidentified; an arbitrary nonnegative regression kernel alone does not guarantee uniqueness. At $p=0$, provided ${\sum }_i{w}_i(x)>0$, the Nadaraya–Watson estimator isMultiplying every weight by $1/h$ changes neither fit.

For the uniform kernel, let $I_x=\{i:|i/n-x|\le h\}$ and $N_x=|I_x|$. Then ${\hat{m}}_h(x)=N_x^{-1}{\sum }_{i\in I_x}{Y}_i$. For $0<h<1/2$ and $x\in [h,1-h]$, the interval $[x-h,x+h]$ is inside $[0,1]$, and the regular design hasThe same lower bound holds when the interval starts at zero, despite the absence of the design point zero. In particular the Nadaraya–Watson estimator is defined throughout the integration interval.

For every Lipschitz continuous mean function in ${\Theta }_L$, the bias of an estimator and variance satisfyHere the variance uses the independence of the errors in the fixed-design nonparametric regression model. The bias-variance decomposition of mean squared error and Tonelli theorem give a slightly stronger integrated mean squared error bound than required:ThusThe printed integration interval represents a nonnegative risk function for $h\le 1/2$. For $1/2<h\le 1$, the written integral has reversed limits and is nonpositive, making the displayed upper bound trivial; it should not be interpreted as an integrated mean squared error in that range. For $h>1$, evaluation outside $[0,1]$ would additionally require a specified extension of the mean function. The subsequent optimization restricts to $h<1/3$.

Minimizing the stated right side givesFor sufficiently large $n$, $n{h}_{\ast }\ge 1$ and $h_{\ast }<1/3$. For example take $n_0=1+\lceil \max \{1,2L/(\sqrt{3}\sigma ),81{\sigma }^2/(4L^2)\}\rceil$. Therefore the upper bound holds with

For the lower bound, the PDF has the mean squared error ${\mathbb{E}}_m[(\tilde{m}(x_0)-m(x_0){)}^2]$; the TeX erroneously moves the square outside the expected value. The PDF's suggested smooth bump function is $\exp (-1/(1-t^2))1_{\{|t|\le 1\}}$, with value zero at the endpoints, rather than the corrupted TeX formula. A triangular alternative suffices because only Lipschitz continuity is needed.

The Le Cam two-point lemma says that for two data laws $P_0,P_1$ with parameter separation $\Delta =|{\theta }_1-{\theta }_0|$, every estimator $T$ satisfiesIndeed, classify according to the nearer parameter value. Misclassification incurs squared-error loss at least ${\Delta }^2/4$, and the sum of the two error probabilities is at least one minus the total variation distance. Averaging the two risk functions proves the stated form.

Let $d=\min (x_0,1-x_0)>0$, $m_0=0$, and use the two-point lower bound with a triangular bumpBoth functions are in ${\Theta }_L$, and $\Delta =Lh$. Assume first that $h\le d$ and $nh\ge 1$. At most $2nh+1$ design points meet the bump's support. The Kullback-Leibler divergence between normal distributions and chain rule for relative entropy giveThe Pinsker's inequality therefore gives $\Vert P_1-P_0{\Vert }_{\mathrm{TV}}\le \sqrt{3/32}<1/2$. The Le Cam two-point lemma yields the pointwise minimax rate for Lipschitz regressionThis universal constant is valid for $n\ge N:=\lceil \max \{1,{\sigma }^2/(8L^2{d}^3),2\sqrt{2}L/\sigma \}\rceil$.

If the printed lower bound is read as applying to all positive integers $n$, it also holds with a smaller constant depending on the fixed $L,\sigma ,x_0$. For $n<N$, use the constant mean functions $m_0=0$ and $m_1=\sigma /\sqrt{n}$. They have Kullback-Leibler divergence $1/2$, so Pinsker's inequality and the Le Cam two-point lemma give minimax risk at least ${\sigma }^2/(16n)$. Combining the two ranges gives the explicit all-$n$ choiceThe universal asymptotic constant and this parameter-dependent all-$n$ constant are distinct conventions; the latter supplies the assertion without inserting an unstated large-$n$ restriction.

At a target $x$, the degree-$p$ local polynomial regression estimate minimizesand ${\hat{m}}_n(x;p,h,K)={\hat{b}}_0$. Let $R_x$ have row
$(1,x_i-x,\dots ,(x_i-x{)}^p)$ and let $W_x$ be diagonal with the kernel weights. Assume the local polynomial Gram matrix $R_x^T{W}_x{R}_x$ is positive definite. The weighted normal equations giveso it is a linear estimator in nonparametric regression.

DefineInverting the two-by-two Gram matrix for $p=1$ yields

Now put $M=\min \{n,\lfloor nh\rfloor \}$. At $x=0$ and for the uniform kernel,For the usual bandwidth range $0<h\le 1$, $M=\lfloor nh\rfloor$. The elementary power-sum formulas and $|M-nh|\le 1$ show, for $r=0,1,2,3$, thatThe limiting moment determinant isFor $nh\ge 32$, the displayed errors can consume at most a fixed fraction of this value, sofor a universal $c>0$ and $0<h\le 1$.

As printed, the claim “for all $h>0$” cannot hold for this design: if $h>1$, all $n$ points receive weight $1/2$, each $s_r=O(h^{-1})$, and the determinant is $O(h^{-2})$, not bounded below by a positive multiple of $h^2$. The standard bandwidth restriction $h\le 1$ is therefore necessary for that intermediate assertion. The final bias bound remains harmless for $h>1$, since $h^2\ge 1$ and the finite design response means are uniformly bounded.

The polynomial reproduction property of local polynomial regression makes the local-linear weights reproduce both $1$ and $u$. Writefor $0\le x_i\le \min (h,1)$. The reproduced terms have zero bias. Using $s_r=O(h^r)$, the numerator contributed by the remainders is bounded byDivision by the determinant lower bound givesThe degree-zero Nadaraya–Watson estimator reproduces constants but not linear functions. The term $x_i$ therefore remains, and its boundary bias is bounded by $Ch$; this first-order boundary bias is the improvement that local linear fitting removes.

PutThe degree-$p$ local polynomial estimator is ${\hat{m}}_n(x)={\hat{b}}_0$, whereDefine the local polynomial Gram matrixAssume that it is positive definite, and writeThe normal equations then giveThus the estimator is a linear estimator in nonparametric regression, and the displayed $w_{p,i}$ are its effective kernel weights.

If $R$ is a polynomial of degree at most $p$, then $R(x+hu)=c^T{r}_p(u)$ for some $c$. Feeding $Y_i=R(x_i)$ into the weighted least-squares problem gives the exact zero-residual fit $c$, whose intercept is $R(x)$. Hence the polynomial reproduction property of local polynomial regression is

Let ${\beta }_0=\lceil \beta \rceil -1$. The Hölder class $\mathcal{H}(\beta ,L)$ consists of functions with derivatives through order ${\beta }_0$ bounded by $L$ andThe subclass $\tilde{\mathcal{H}}(\beta ,L)$ additionally makes every derivative of order below ${\beta }_0$ $L$-Lipschitz.

Suppose $p<{\beta }_0$. Taylor's theorem and that additional Lipschitz condition give, for the degree-$p$ Taylor polynomial $T_p$ at $x$,On the support of $K_i$, $|u_i|\le 1$, soFor the regular design $x_i=i/n$, at most $2nh+1\le 4nh$ points satisfy $|x_i-x|\le h$ when $h\ge 1/(2n)$. Polynomial reproduction cancels $T_p$, and thereforeThus the universal exponent in the question is $\gamma =1$, with the displayed choice of $C({\lambda }_0,p,L,K)$.

Write $t_i=(x_i-x)/h$, $r_p(t)=(1,t,\dots ,t^p{)}^T$, and defineThe degree-$p$ local polynomial regression fit minimizesAssume its local polynomial Gram matrixis positive definite. The normal equations then give

Because the polynomial is written in the scaled coordinate, the local polynomial derivative estimator is ${\hat{m}}_n^{\prime }(x)={\hat{\beta }}_1/h$. If $e_1=(0,1,0,\dots ,0{)}^T$, thenFor a polynomial $R$ of degree at most $p$, the local least-squares fit to $Y_i=R(x_i)$ is exactly the Taylor polynomial $R(x+ht)$, so its scaled linear coefficient is $h{R}^{\prime }(x)$. Therefore the polynomial reproduction property of local polynomial regression gives

Positive definiteness and the fixed finite-dimensional basis provide a number ${\lambda }_0>0$ such thatSince $K(t_i)=0$ outside $|x_i-x|\le h$,and

The regular design has at most $2nh+2\le 4nh$ points in this window when $h\ge 1/(2n)$. Since the errors are independent with variance at most ${\sigma }^2$,

Finally let ${\beta }_0=\lceil \beta \rceil -1$ and take the degree-${\beta }_0$ Taylor polynomial $R$ of $m$ at $x$. The Hölder class remainder satisfiesPolynomial reproduction removes $R$ from the bias. On the kernel window the remainder is at most $L{h}^{\beta }/{\beta }_0!$, so the weight-sum bound gives

A linear estimator in nonparametric regression at $x_0$ has the form ${\hat{m}}_n(x_0)={\sum }_i{W}_i(x_0)Y_i$, where the weights may depend on the design, $x_0$, $p$, $h$, and $K$, but not on the responses. Putand define the local polynomial Gram matrixWhen ${\lambda }_0>0$, the weighted least squares normal equations have the unique solutionTherefore ${\hat{m}}_n(x_0)={\sum }_i{W}_i{Y}_i$, where the effective kernel weight is