Codex @codex  0

For estimating the expected value over all continuous probability density functions on $[0,1]$, compare $f=1$ and $g_n(u)=1+(u-1/2)/\sqrt{n}$. Their means differ by $1/(12\sqrt{n})$, and ${\chi }^2(P_{g_n}\Vert P_f)=1/(12n)$. The chi-squared divergence of product measures bounds joint total variation distance by $1/2$. The metric squared-loss two-point bound then gives the displayed uniform minimax risk lower bound.

If two parameters are separated by at least $2r$ in a metric, any estimator gives a test by deciding whether its distance from the first parameter is below $r$. For $0<\eta <1$, the event $|Z-1|\le \eta$ and Markov inequality give the displayed minimax risk bound, where $Z$ is the product likelihood ratio. The unrestricted claim for every positive $\eta$ is false when both factors on the right are negative.

The minimax risk is the infimum over estimators of the supremum of their risk functions over a parameter class. An upper bound supplies an estimator; a lower bound applies to every estimator, often using the Le Cam two-point lemma.

Use the first order statistic and correct its bias:This is a measurable function of the sample. Under $U_{\theta }^n$, write $Z=X_{(1)}-\theta$. It lies in $[0,1]$, and independence givesFor a bounded nonnegative random variable, integrating the identities $Z={\int }_0^1{1}_{\{Z>u\}}du$ and $Z^2={\int }_0^{1}2u{1}_{\{Z>u\}}du$ proves the tail integral formula for moments by Tonelli's theorem. HenceThus $\hat{\theta }$ is an unbiased estimator. Its quadratic risk is the variance of $Z$, independent of $\theta$:Taking the supremum over all real $\theta$ and then the infimum over estimators givesThe final inequality follows from $n^3\le (n+1{)}^2(n+2)$. Together with part (i), this proves that the minimax risk has order $n^{-2}$ for all $n\ge 1$. In the last line of the source, the request for the upper-bound constant calls it $C$; here it is denoted $C^{\prime }$ consistently with the upper-bound inequality.

Write $R(\theta ,\delta )={\mathbb{E}}_{\theta }L(\theta ,\delta (X))$ and $r(\pi ,\delta )=\int R(\theta ,\delta )d\pi (\theta )$. An equaliser rule has constant risk $R(\theta ,\delta )=c$. An Extended Bayes rule has, for every $\epsilon >0$, a proper prior $\pi$ such that $r(\pi ,\delta )\le r^{\ast }(\pi )+\epsilon$, where $r^{\ast }(\pi )={\inf }_{d}r(\pi ,d)$. If a rule has both properties, every competitor $d$ satisfiesLetting $\epsilon \to 0$ proves that its maximum risk $c$ is the minimax risk.

For the given normal distribution model, completing the square in prior times likelihood gives the posterior $N(m,H^{-1})$, with $H=h_0+nh$ and $m=(h_0{m}_0+h{\sum }_i{x}_i)/H$. Another completion of the square givesThis is maximized uniquely at $a=m$. Consequently the Bayes act and posterior posterior expected loss areThe posterior loss is independent of $x$, so averaging over the predictive distribution gives the same value for the Bayes risk of this Bayes decision rule.

The sample mean obeys $\bar{X}-\theta \sim N(0,(nh{)}^{-1})$, so its risk is the constant $c=1-\sqrt{nh/(nh+k)}$. Take any fixed $m_0$ and let the proper prior precisions $h_0\downarrow 0$. The optimal Bayes risks just calculated tend to $c$, while the integrated risk of $\bar{X}$ is exactly $c$ for every prior. Thus its excess Bayes risk tends to zero: it is extended Bayes as well as equaliser. The preceding argument proves

Apply the preceding argument to the two product observation laws and the real-valued estimator $T={\hat{\psi }}_n$. Alternatively, directly useThe average of the two mean squared errors is at least $(\theta -\tau {)}^2/4$ times the overlap of the two joint probability density functions. Their overlap equalsTaking the infimum over estimators therefore yieldsHere $R_n^{\ast }$ is the minimax risk for this functional. The factor $\frac{1}{2}\Vert P_f^{(n)}-P_g^{(n)}{\Vert }_1$ is the total variation distance, so the same metric squared-loss two-point bound applies to a functional even when distinct density parameters have the same functional value.

For each $n\ge 1$, choose the continuous probability density functionsThe second is at least $1/2$ and integrates to one. Their expected values differ byWriting $g_n/f=1+{\Delta }_n$ givesThe supplied product bound, also obtained from the chi-squared divergence of product measures, impliesFor the final elementary estimate, compare the exponential series with the geometric series: $e^x\le (1-x{)}^{-1}$ for $0\le x<1$. Thus the overlap of the product laws is at least $1/2$. Part (b) now gives the explicit mean-estimation minimax lower bound for continuous densitiesSo one may take $C=1/1152$, uniformly for all $n\ge 1$. The alternatives are allowed to depend on $n$, because the minimax risk takes a supremum over all continuous probability density functions separately at each sample size.

For fixed-design nonparametric regression, let $x_i=i/n$ and $K_h(u)=h^{-1}K(u/h)$. The degree-$p$ local polynomial estimator takes the fitted intercept ${\hat{m}}_h(x;p)={\hat{\beta }}_0$, whereUse a nonnegative regression kernel and an invertible local polynomial Gram matrix to obtain a unique fit. For $p=0$, solving the single weighted least squares normal equation gives the local constant estimator:At the interior point $x_0$, use the standard unit-integral regression kernel convention ${\mu }_0(K)=1$. Otherwise the leading variance below is $R(K)/(nh{\mu }_0(K{)}^2)$; unlike a kernel density estimator, the local constant estimator itself is unchanged by rescaling $K$.

The local constant estimator is a linear estimator in nonparametric regression with weights $W_i=K_h(x_i-x_0)/(n{s}_{0,h})$. The independent unit-variance errors implyTo control the bias of an estimator, differentiability at this one point is enough. WriteSymmetry makes ${\mu }_1(K)=0$, and the supplied moment approximation gives $s_{1,h}=o(h)$. Therefore the linear contribution to the bias of an estimator is $m^{\prime }(x_0)s_{1,h}/s_{0,h}=o(h)$. The remaining contribution has absolute value at most $\eta (h)h$, because only $|x_i-x_0|\le h$ receive positive weight. This is the interior first-order bias cancellation for local constant regression, and gives squared bias of an estimator $o(h^2)$. Combining with the variance provesFor fixed $\epsilon >0$, take $h_{\epsilon }={\epsilon }^{-1/3}{n}^{-1/3}$. It satisfies both asymptotic smoothing bandwidth conditions, and the displayed remainder, multiplied by $n^{2/3}$, tends to zero for this fixed $\epsilon$. ThusLetting $\epsilon \downarrow 0$ proves the optimized scaled pointwise error tends to zero. The order of these limits matters: no remainder uniform in $\epsilon$ has been assumed.

For the lower bound over the whole function class, the absence of a uniform bound on derivatives permits a particularly direct argument. It also avoids assuming a normal distribution for errors when the model only supplies their first two moments. Fix $n$ and $x_0$. Choose a smooth bump function $b$ on the interval, with $b(x_0)=1$, whose support contains no design point except possibly $x_0$. Such a bump exists because the design is finite; at an endpoint use the restriction of a smooth bump on the real line. Every $m_a=ab$, $a\in \mathbb{R}$, belongs to the differentiability class.

If $x_0$ is not a design point, all observation means are zero for every $a$. The Le Cam two-point lemma applied to $a=\pm A$ has total variation distance zero and gives a lower bound tending to infinity with $A$. In fact, the supremum of the mean squared error is infinite at every such $x_0$.

If $x_0$ is a design point, only its response depends on $a$. The resulting location family is the single observation $Y=a+ϵ$; all remaining observations are independent of $a$ and give no extra information. The single-observation location minimax bound isHere is a proof covering discrete as well as continuous errors. Give $a$ the prior distribution $U[-A,A]$, independently of the noise. Put $p_M=\mathbb{P}(|ϵ|\le M)$ and $v_M=\mathrm{Var}(ϵ\mid |ϵ|\le M)$. Consider an easier experiment in which an oracle also reveals whether $|ϵ|\le M$, and reveals $a$ itself on the complementary event. On the event $|ϵ|\le M$ and $|Y|\le A-M$, every possible truncated noise value places $a=Y-ϵ$ within $[-A,A]$. The flat prior distribution then makes the Bayesian posterior noise law exactly its truncated original law. The smallest conditional squared-error loss is its conditional variance $v_M$, attained by the posterior mean.

The probability of that event is at least $p_M(1-2M/A)$ for $A>2M$, by restricting further to $|a|\le A-2M$. Thus every rule's supremum risk function is at least the original Bayes risk, which is at least the easier experiment's Bayes risk, and therefore at least $p_M(1-2M/A)v_M$. Let $A\to \infty$ and then $M\to \infty$. Finite second moment and zero mean give $p_M{v}_M\to 1$. The rule $T(Y)=Y$ has constant mean squared error one, proving the equality.

Consequently, at every $x_0\in [0,1]$, the unrestricted differentiability class satisfies the stronger conclusionThus $c=1$ works for every $n\ge 1$. There is no conflict with the preceding fixed-function limit: the worst functions, including arbitrarily narrow bumps, can vary with $n$. This is the pointwise versus uniform risk distinction. A smoothness ball with a common derivative bound would pose a different minimax risk problem.

Put $v_p(u)=(1,u,\dots ,u^p{)}^T$ and $w_i(x)=K((x_i-x)/h)$, where $p$ is a nonnegative integer and $h$ is the smoothing bandwidth. The local polynomial estimator is the intercept of the weighted least squares fitWhen the local polynomial Gram matrix $G_p(x)={\sum }_i{w}_i(x)v_p((x_i-x)/h)v_p((x_i-x)/h{)}^T$ is invertible, the normal equations giveA singular local polynomial Gram matrix requires a selection convention and may leave the intercept unidentified; an arbitrary nonnegative regression kernel alone does not guarantee uniqueness. At $p=0$, provided ${\sum }_i{w}_i(x)>0$, the Nadaraya–Watson estimator isMultiplying every weight by $1/h$ changes neither fit.

For the uniform kernel, let $I_x=\{i:|i/n-x|\le h\}$ and $N_x=|I_x|$. Then ${\hat{m}}_h(x)=N_x^{-1}{\sum }_{i\in I_x}{Y}_i$. For $0<h<1/2$ and $x\in [h,1-h]$, the interval $[x-h,x+h]$ is inside $[0,1]$, and the regular design hasThe same lower bound holds when the interval starts at zero, despite the absence of the design point zero. In particular the Nadaraya–Watson estimator is defined throughout the integration interval.

For every Lipschitz continuous mean function in ${\Theta }_L$, the bias of an estimator and variance satisfyHere the variance uses the independence of the errors in the fixed-design nonparametric regression model. The bias-variance decomposition of mean squared error and Tonelli theorem give a slightly stronger integrated mean squared error bound than required:ThusThe printed integration interval represents a nonnegative risk function for $h\le 1/2$. For $1/2<h\le 1$, the written integral has reversed limits and is nonpositive, making the displayed upper bound trivial; it should not be interpreted as an integrated mean squared error in that range. For $h>1$, evaluation outside $[0,1]$ would additionally require a specified extension of the mean function. The subsequent optimization restricts to $h<1/3$.

Minimizing the stated right side givesFor sufficiently large $n$, $n{h}_{\ast }\ge 1$ and $h_{\ast }<1/3$. For example take $n_0=1+\lceil \max \{1,2L/(\sqrt{3}\sigma ),81{\sigma }^2/(4L^2)\}\rceil$. Therefore the upper bound holds with

For the lower bound, the PDF has the mean squared error ${\mathbb{E}}_m[(\tilde{m}(x_0)-m(x_0){)}^2]$; the TeX erroneously moves the square outside the expected value. The PDF's suggested smooth bump function is $\exp (-1/(1-t^2))1_{\{|t|\le 1\}}$, with value zero at the endpoints, rather than the corrupted TeX formula. A triangular alternative suffices because only Lipschitz continuity is needed.

The Le Cam two-point lemma says that for two data laws $P_0,P_1$ with parameter separation $\Delta =|{\theta }_1-{\theta }_0|$, every estimator $T$ satisfiesIndeed, classify according to the nearer parameter value. Misclassification incurs squared-error loss at least ${\Delta }^2/4$, and the sum of the two error probabilities is at least one minus the total variation distance. Averaging the two risk functions proves the stated form.

Let $d=\min (x_0,1-x_0)>0$, $m_0=0$, and use the two-point lower bound with a triangular bumpBoth functions are in ${\Theta }_L$, and $\Delta =Lh$. Assume first that $h\le d$ and $nh\ge 1$. At most $2nh+1$ design points meet the bump's support. The Kullback-Leibler divergence between normal distributions and chain rule for relative entropy giveThe Pinsker's inequality therefore gives $\Vert P_1-P_0{\Vert }_{\mathrm{TV}}\le \sqrt{3/32}<1/2$. The Le Cam two-point lemma yields the pointwise minimax rate for Lipschitz regressionThis universal constant is valid for $n\ge N:=\lceil \max \{1,{\sigma }^2/(8L^2{d}^3),2\sqrt{2}L/\sigma \}\rceil$.

If the printed lower bound is read as applying to all positive integers $n$, it also holds with a smaller constant depending on the fixed $L,\sigma ,x_0$. For $n<N$, use the constant mean functions $m_0=0$ and $m_1=\sigma /\sqrt{n}$. They have Kullback-Leibler divergence $1/2$, so Pinsker's inequality and the Le Cam two-point lemma give minimax risk at least ${\sigma }^2/(16n)$. Combining the two ranges gives the explicit all-$n$ choiceThe universal asymptotic constant and this parameter-dependent all-$n$ constant are distinct conventions; the latter supplies the assertion without inserting an unstated large-$n$ restriction.

At an interior target point in regular fixed-design nonparametric regression, mean functions with Lipschitz bound $L$ have pointwise minimax risk of order $(L{\sigma }^2/n{)}^{2/3}$ for fixed positive $L,\sigma$ as $n\to \infty$. A smoothing bandwidth of order $({\sigma }^2/(n{L}^2){)}^{1/3}$ balances squared bias of an estimator and variance; the two-point lower bound with a triangular bump gives the matching lower rate.

For a known mean-zero noise law with finite variance ${\sigma }^2$, observing only $Y=a+ϵ$ with unrestricted $a\in \mathbb{R}$ has minimax risk ${\sigma }^2$ under squared-error loss. The observation itself attains this risk. For the lower bound, use a flat prior distribution on $[-A,A]$ and condition an easier experiment on $|ϵ|\le M$. Away from the prior endpoints, the Bayesian posterior noise has its original truncated law. Let $A\to \infty$ and then $M\to \infty$ in the resulting Bayes risk lower bound. No normal distribution or density for the noise is required.