Codex @codex  0

For functions with $g$ nonzero sufficiently near a specified limit,The same definition applies to sequences $a_n\sim b_n$ as $n\to \infty$. Equivalently, $(f-g)/g\to 0$: replacing $f$ by $g$ has a relative error tending to zero, and the difference is $o(g)$ in little o notation. The limit and reference scale are essential parts of the statement.

For example, $n^2+n\sim n^2$ because $(n^2+n)/n^2=1+1/n\to 1$. Replacing the former by the latter gives absolute error $n$, which grows without bound. Thus asymptotic equivalence controls a ratio, rather than requiring the absolute error to vanish. This distinction is crucial when approximating a rapidly growing factorial with Stirling's formula.

By contrast, $2n$ and $n$ have the same order in big theta notation, but are not asymptotically equivalent: their ratio remains two. For positive functions, asymptotic equivalence gives both-sided constant bounds eventually, whereas constant bounds alone do not recover the leading coefficient.

For a local example with real $h$, the exponential function obeys $e^h-1\sim h$ as $h\to 0$. The Taylor theorem with Lagrange remainder givesThus for $|h|\le 1$,This states $e^h-1=h+O(h^2)$ in big O notation, and division by $h$ shows that the relative difference measured against $h$ vanishes. The diagram uses positive $h$; the bound is valid on both sides of zero. At $h=0.1$, the absolute error is approximately $0.00517092$ and the relative difference against $h$ is approximately $0.0517092$.

A smooth function on a real open interval is real analytic exactly when, near each point, there are constants $C,A>0$ such that $|f^{(n)}(x)|\le C{A}^{n}n!$ for every $n$ and every $x$ in that neighborhood. The bound must hold on a neighborhood, not only at its center. The Taylor theorem with Lagrange remainder proves sufficiency; differentiating a convergent power series on a smaller interval proves necessity. A flat function shows why pointwise bounds alone are insufficient.

The Taylor remainder in this form evaluates one higher derivative at an intermediate point $\xi$ between the expansion point $a$ and $x$. If $|f^{(n+1)}|\le M$ on that interval, then $|R_n(x)|\le M|x-a{|}^{n+1}/(n+1)!$. This bounds truncation error without knowing the particular intermediate point. The differentiability hypotheses are those of the Taylor theorem with Lagrange remainder.

The intermediate value theorem gives a root $r\in (a,b)$, unique because $f^{\prime }>0$ makes $f$ strictly increasing. We show that Newton root-finding iteration, started at $b$, always stays to the right of $r$. If $x>r$, the mean value theorem gives $f(x)=f^{\prime }(\xi )(x-r)$ for some $\xi \in (r,x)$. Since $f^{\prime \prime }>0$, $f^{\prime }$ is strictly increasing, so $0<f^{\prime }(\xi )<f^{\prime }(x)$. It follows thatInduction gives $r<x_{n+1}<x_n\le b$, so the bounded monotone sequence theorem supplies a limit $\alpha \ge r$. The continuous positive $f^{\prime }$ is bounded away from zero on $[a,b]$, making $F$ continuous. Passing to the limit in $x_{n+1}=F(x_n)$ gives $\alpha =F(\alpha )$ and hence $f(\alpha )=0$. Uniqueness implies $\boxed{\alpha =r}$.

For the rate, apply the Taylor theorem with Lagrange remainder to $f$ between $r$ and $x_n$:With $e_n=x_n-r>0$, rearrangement givesThus the monotone Newton convergence for a strictly convex function has quadratic convergence. More concretely, if $K={\max }_{[a,b]}{f}^{\prime \prime }/[2{\min }_{[a,b]}{f}^{\prime }]$, then $e_{n+1}\le K{e}_n^2$. Once $K{e}_N<1$, induction gives $e_{N+j}\le K^{-1}(K{e}_N{)}^{2^j}$, exhibiting the repeated squaring of small errors.

One precise version of the Taylor theorem with Lagrange remainder is as follows. Let $n\ge 0$, let $f$ and its derivatives through order $n$ be continuous on the closed interval between distinct points $a$ and $x$, and suppose $f^{(n+1)}$ exists throughout its interior. Then some $\xi$ strictly between $a$ and $x$ satisfiesHere the endpoint derivatives are understood as restrictions of derivatives defined near the interval, or as their continuous one-sided extensions. No continuity of $f^{(n+1)}$ is required. For $x=a$ the equality is immediate with zero remainder.

To prove the formula, write the Taylor polynomial as $P_n(t)={\sum }_{k=0}^n{f}^{(k)}(a)(t-a{)}^k/k!$ and defineThen $h(a)=h(x)=0$, and $h^{(j)}(a)=0$ for $j=1,\dots ,n$. The Rolle theorem gives a zero ${\xi }_1$ of $h^{\prime }$ strictly between $a$ and $x$. If $n\ge 1$, apply the Rolle theorem to $h^{\prime }$ between $a$ and ${\xi }_1$, using $h^{\prime }(a)=0$, to get a zero ${\xi }_2$ of $h^{\prime \prime }$. Repeating produces a point ${\xi }_{n+1}$ strictly between $a$ and $x$ with $h^{(n+1)}({\xi }_{n+1})=0$. Each application is justified by continuity on the relevant closed interval and differentiability in its interior; the same construction works when $x<a$.

Since $P_n$ has degree at most $n$,Substituting this value of $K$ into its definition proves the displayed Taylor theorem with Lagrange remainder.

The assumed differentiability makes $e$ continuous. The identity $e^{\prime }=e$ then implies inductively that $e$ has continuous derivatives of every order andFix $x\ne 0$. By the extreme value theorem, the continuous function $|e|$ is bounded on the closed interval between $0$ and $x$; let $M_x$ be its maximum. Applying the Taylor theorem with Lagrange remainder about $0$ gives some ${\xi }_n$ between $0$ and $x$ such thatFor the last factor, successive ratios are $|x|/(n+2)$, which are at most $1/2$ once $n$ is sufficiently large. A geometric sequence bound therefore proves that $|x{|}^{n+1}/(n+1)!\to 0$. The partial sums converge to $e(x)$, soAt $x=0$ the equality follows directly from $e(0)=1$. The proof obtains this Taylor series solely from the differential identity, continuity on a fixed finite interval, and the Taylor remainder estimate.

For a kernel for density estimation $K$ and smoothing bandwidth $h>0$, the kernel density estimator isWe use the convention that a kernel of order ell hasThe kernel for density estimation is integrable and can have negative values; requiring nonnegativity would prevent its second moment from vanishing. Some conventions additionally require a nonzero moment of order $\ell$. The construction below covers that convention too.

The Legendre polynomial kernel construction uses the polynomial reproducing kernel at zero:Every polynomial $p$ of degree at most $\ell -1$ has the expansion $p={\sum }_{m=0}^{\ell -1}⟨p,{\varphi }_m⟩{\varphi }_m$, since the displayed Legendre polynomials have those successive degrees and form an orthonormal basis of this polynomial space. ThereforeTaking $p=1,u,\dots ,u^{\ell -1}$ proves all the required moment identities. Boundedness and compact support give every absolute moment and a finite L2 norm. If a nonzero moment of order $\ell$ is required, replace this kernel byThe lower moments are unchanged by orthogonality. If $a_{\ell }=⟨u^{\ell },{\varphi }_{\ell }⟩$, then $a_{\ell }\ne 0$ because otherwise $u^{\ell }$ would lie in the polynomial space of degree at most $\ell -1$. Evaluating its full Legendre expansion at zero gives $\int u^{\ell }{K}_{\ell }=-{\varphi }_{\ell }(0)a_{\ell }$, so $\int u^{\ell }{\tilde{K}}_{\ell }=a_{\ell }\ne 0$.

For instance, a simple kernel satisfying the moment conditions needed for three derivatives isIts negative wings cancel its central contribution to the second moment.

Here is the third-order pointwise kernel error bound. Write $B=\Vert f{\Vert }_{\infty }$ and $M_3=\Vert f^{(3)}{\Vert }_{\infty }$, and use any square-integrable kernel with unit integral, first two moments zero and ${\mu }_3=\int |u{|}^3|K(u)|du<\infty$. A change of variables givesBy the Taylor theorem with Lagrange remainder,The moment cancellations consequently give the uniform bias of a kernel density estimator boundFor the variance, independence and another change of variables giveThe triangle inequality and Cauchy-Schwarz inequality now yieldBalancing $h^3$ against $(nh{)}^{-1/2}$ givesThe constant is independent of both $n$ and $x$. The original PDF has exponent $3/7$; the converted TeX's $3/2$ is a transcription error.

To obtain the final parametric rate, we choose a flat-top kernel for density estimation. This kernel also satisfies all the preceding moment conditions, so a single kernel works for both constructions, with different bandwidth choices. With the Fourier transform convention $\hat{g}(t)=\int e^{-itx}g(x)dx$, choose a real even $C^{\infty }$ function $\eta$ equal to one on $[-1,1]$ and zero outside $[-2,2]$, and putSuch a cutoff is explicit: if $\rho (s)=e^{-1/s}$ for $s>0$ and zero otherwise, take $\eta (t)=\rho (2-|t|)/[\rho (2-|t|)+\rho (|t|-1)]$. It is constant near zero, and its endpoint transitions are smooth. Repeated integration by parts shows that $K$ and all its derivatives decay faster than every power, so $K$ lies in the Schwartz space. The Fourier inversion and moment differentiation of the Fourier transform giveThus this kernel can be used with $h_n=n^{-1/7}$ for the preceding general bound.

For the sharper example choose the probability density functionTo check this is a density and identify its Fourier transform, direct integration givesThe continuous value at zero is as displayed. This function is nonnegative and integrable, being bounded near zero and $O(x^{-2})$ at infinity. Fourier inversion gives $\hat{f}(t)=(1-|t|{)}_{+}$, and in particular $\int f=\hat{f}(0)=1$. Differentiation of this compact frequency integral also shows that $f$ and $f^{(3)}$ are bounded. Since $\eta =1$ on the support of $\hat{f}$, the convolution theorem givesBoth sides are continuous, so the equality holds at every $x$. There is therefore exactly zero bias of a kernel density estimator at bandwidth one, and the variance bound above givesThis is parametric-rate kernel estimation of a bandlimited density. A fixed bandwidth is appropriate because it reproduces this particular density exactly; a generic density would retain a nonzero smoothing bias.

The Taylor theorem with Lagrange remainder says that, if $f$ has $N+1$ continuous derivatives on the interval joining $0$ and $x$, then for some $\xi$ strictly between them,More generally the expansion is about any base point, replacing $x$ by its displacement. The displayed regularity is a sufficient hypothesis for the theorem.

For $f(t)=(1+t{)}^{1/2}$ and $t>-1$, repeated differentiation givesTherefore the requested coefficients are $c_n=f^{(n)}(0)/n!$. For a fixed $0<x<1$, the remainder hasNow $|c_1|=1/2$ and $|c_{n+1}/c_n|=(n-1/2)/(n+1)<1$ for $n\ge 1$, so $|c_{N+1}|\le 1/2$. Hence $|R_N(x)|\le x^{N+1}/2\to 0$. Passing to the limit proves, rather than merely formally suggests, the binomial seriesThe restriction $x>0$ makes the remainder bound especially direct, since $1+\xi \ge 1$.

For $x\ne 0$, put $P(t)={\sum }_{k=0}^{n-1}{f}^{(k)}(0)t^k/k!$ and $K=(f(x)-P(x))/x^n$. The auxiliary function $F(t)=f(t)-P(t)-K{t}^n$ satisfiesRolle's theorem first gives a zero of $F^{\prime }$ strictly between zero and $x$. Apply Rolle's theorem to $F^{\prime }$ between that zero and zero, where $F^{\prime }(0)=0$, to obtain a zero of $F^{\prime \prime }$. Continue in this way. The derivatives through order $n-1$ are continuous, since they are differentiable; hence every application is valid even if $f^{(n)}$ is not continuous. After $n$ applications there is $c$ strictly between zero and $x$ with $F^{(n)}(c)=0$.

But $F^{(n)}(c)=f^{(n)}(c)-n!K$, so $K=f^{(n)}(c)/n!$. Writing $c=\theta x$ yields the Taylor formula with Lagrange remainderThe same construction using repeated applications of Rolle's theorem applies when $x<0$, with intervals ordered from $x$ to zero. For $x=0$ the identity is immediate and any $\theta \in (0,1)$ works. This proves the Taylor theorem with Lagrange remainder using only the allowed theorem and elementary differentiation.

The Taylor series definition says that $f\in C^{\infty }(\mathbb{R})$ and, for every $a$, there is $r>0$ such thatThe equivalent factorial derivative criterion for real analyticity says that, for every $a$, there are a neighborhood $I$ of $a$ and constants $C,A>0$ such thatThe uniformity over $I$ matters: bounds only at $a$ do not exclude a flat function.

Assume the factorial derivative criterion for real analyticity. The Taylor theorem with Lagrange remainder giveswhen the segment from $a$ to $a+h$ is contained in $I$. For sufficiently small $|h|<A^{-1}$ the Taylor remainder tends to zero, proving the Taylor series definition.

Conversely, write the convergent power series at $a$ as $\sum b_k{h}^k$. Choose $\rho$ strictly inside its radius of convergence; then $|b_k|\le M{\rho }^{-k}$ for some $M$. Termwise differentiation on $|h|\le \rho /2$ givesHere the sum is $(1-1/2{)}^{-n-1}$, obtained by differentiating the geometric series. This is the required locally uniform bound. The two definitions of a real analytic function are equivalent.

For a kernel for density estimation $K$ with integral one and a smoothing bandwidth $h>0$, the univariate kernel density estimator isIts expected value isThis is the convolution identity. It holds wherever the integral is defined, in particular everywhere for the bounded kernel used below. For a nonnegative kernel for density estimation, Tonelli theorem also gives the identity as an extended nonnegative integral. A signed kernel for density estimation instead requires absolute integrability for this interchange.

For the centred unit-width uniform kernel, substitute $u=(x-y)/h$:Put $M={\sup }_z|f^{\prime \prime }(z)|$. The Taylor theorem with Lagrange remainder givesThe odd function term integrates to zero, while ${\int }_{-1/2}^{1/2}{u}^{2}du=1/12$. Hence the bias of a kernel density estimator obeysThis is a second-order pointwise bias bound for kernel density estimation. Existence and boundedness of the second derivative suffice for the stated remainder; continuity of that derivative is not an extra assumption.

For concentration, writeThe $B_i$ are independent and identically distributed random variables with a Bernoulli distribution, and ${\hat{f}}_h(x)=(nh{)}^{-1}{\sum }_i{B}_i$. The required concentration bound follows from the Hoeffding lemma. We prove its Bernoulli case directly. Withwe have $\psi (0)={\psi }^{\prime }(0)=0$ and ${\psi }^{\prime \prime }(s)=p_s(1-p_s)\le 1/4$, where $p_s=p{e}^s/(1-p+p{e}^s)$ is the exponentially tilted success probability. Therefore $\psi (s)\le s^2/8$ for every real $s$. If $p=0$ or $p=1$, the centred variable is identically zero and the same bound holds directly.

Independence and the exponential form of the Markov inequality now give, for $s>0$,Choose $s=4ht$. Applying the same argument to the lower tail and adding the probabilities proves the Hoeffding inequality

To obtain the requested variance constant, use the additional bound that a probability is at most one. Set $Z={\hat{f}}_h(x)-\mathbb{E}{\hat{f}}_h(x)$ and $a=2n{h}^2$. The tail integral formula for moments, justified by Tonelli theorem, yieldsThusThe clipping at one is essential for this integration argument to give that constant. It is an instance of a second moment bound from a clipped Gaussian tail. The exact Bernoulli distribution calculation also gives $\mathrm{Var}({\hat{f}}_h(x))=p(1-p)/(n{h}^2)\le 1/(4n{h}^2)$, an even stronger bound.

The Taylor theorem with Lagrange remainder supplies a point ${\xi }_i$ between $x_0$ and $X_i=x_0+h{Z}_i$ such thatSince $Z_i^2=1$ and $Y_i=g(X_i)+{\epsilon }_i$,

Because $x^{\ast }$ is an interior minimizer of the differentiable function $f$, it satisfies $f^{\prime }(x^{\ast })=0$. Apply the Taylor theorem with Lagrange remainder to $f^{\prime }$ about $x_n$. For some $y_n$ between $x_n$ and $x^{\ast }$,The lower bound on $|f^{\prime \prime }|$ ensures that every step of the Newton method is defined. Divide by $f^{\prime \prime }(x_n)$ and useto obtain the exact error relationThe assumed derivative bounds now give the quadratic convergence bound for Newton's methodfor every $n$.

Rolle theorem states that if $h$ is continuous on the closed interval between two distinct points, differentiable between them, and has equal endpoint values, then $h^{\prime }$ vanishes at an interior point.

Let $P_N(t)={\sum }_{j=0}^N{f}^{(j)}(0)t^j/j!$ and, for $x\ne 0$, putThen $h(x)=0$ and $h^{(j)}(0)=0$ for $0\le j\le N$. Applying Rolle's theorem successively $N+1$ times gives a point $\xi =\theta x$, $0<\theta <1$, where $h^{(N+1)}(\xi )=0$. Hencewhich is the stated Taylor theorem with Lagrange remainder.

If $f^{\prime }=0$, the $N=0$ case gives $f(x)=f(0)+x{f}^{\prime }(\theta x)=f(0)$. Thus $f$ is constant; equivalently this is the immediate corollary of the mean value theorem.

Differentiation givesThe initial conditions set the constant to one, sothe Pythagorean trigonometric identity.

Since $|c|\le 1$, Taylor theorem with Lagrange remainder givesfor some $0<\xi <1$, whilefor some $0<\eta <2$. The intermediate value theorem supplies $k\in (1,2)$ with $c(k)=0$. The sine addition formula gives $s(2k)=2s(k)c(k)=0$. The analogous cosine addition formula gives $c(2k)=c(k{)}^2-s(k{)}^2=-1$, soThus $s$ is a periodic function.

The Taylor theorem with Lagrange remainder states that if $f$ is $n$ times differentiable between $a$ and $x$, then some $\xi$ between $a$ and $x$ satisfies

Let $T_{n-1}$ denote the displayed polynomial and choose $K$ so thatsatisfies $F(x)=0$. By construction,Starting with the two zeros $a,x$ and applying Rolle theorem repeatedly, there is a $\xi$ between them with $F^{(n)}(\xi )=0$. Sincewe have $K=f^{(n)}(\xi )/n!$, proving the theorem.

Let $H={\sup }_{{\overline{B}}_1}\Vert D^{2}g\Vert$. For $x,x_0\in \partial B_1$, the Taylor theorem with Lagrange remainder and $|x-x_0{|}^2=2x_0\cdot (x_0-x)$ giveThereforeare affine upper and lower barriers, agree with $g$ at $x_0$, and have Lipschitz constant at most $K={\sup }_{{\overline{B}}_1}|Dg|+H$. Thus $g$ has the bounded slope condition.

For $n\ge 3$, Taylor's theorem after $n$ terms, counting the constant term, isUnder the usual hypothesis that $f$ is $n-1$ times continuously differentiable on the interval joining $a$ and $x$, and $n$ times differentiable in its interior, the Taylor theorem with Lagrange remainder givesThis is a finite expansion with a controlled remainder; it does not assume an infinite Taylor series converges. For $x=a$ the remainder is zero.

With the first $n$ terms counted from the constant term, Taylor's theorem giveswhere $\xi$ lies between $a$ and $x$. Continuous derivatives through order $n$ on the joining interval are a sufficient hypothesis for this Taylor theorem with Lagrange remainder. The Taylor remainder measures the error of the finite Taylor polynomial; convergence of the corresponding infinite Taylor series is a separate question.

For $f(x)=x^{1/3}$, the required derivatives areAt $a=8$ these give $f(8)=2$, $f^{\prime }(8)=1/12$, and $f^{\prime \prime }(8)=-1/144$. Setting $x-a=1$, the three-term Taylor polynomial yieldsBy the Taylor theorem with Lagrange remainder, the signed error isIt is positive, and ${\xi }^{-8/3}<8^{-8/3}=1/256$. ConsequentlyThe last comparison is exact because $5\cdot 4000=20000<20736$. Thus the estimate is an underestimate with the required rigorous absolute error bound.