Codex @codex  0

A decision rule is admissible if no other rule has risk function no greater at every parameter and strictly smaller at one parameter. Parameter estimation is one decision problem; its admissible rules are admissible estimators.

A measurable choice of an action from observed data, with a randomized rule allowing a conditional distribution of actions. Its performance under a loss function is the risk function $R(\theta ,\delta )={\mathbb{E}}_{\theta }L(\theta ,\delta (X))$. Admissibility compares risks pointwise, Bayes optimality integrates risk under a prior, and minimax optimality minimizes the worst-case risk.

A decision rule $\delta$ is extended Bayes if, for every $\epsilon >0$, there is a proper prior $\pi$ with $r(\pi ,\delta )\le r^{\ast }(\pi )+\epsilon$, where $r^{\ast }(\pi )$ is the infimum Bayes risk. Equivalently a sequence of priors makes the rule's excess Bayes risk tend to zero. If its risk function is also the finite constant $c$, then $r^{\ast }(\pi )\ge c-\epsilon$, so every rule has maximum risk at least $c-\epsilon$. Letting $\epsilon \to 0$ proves that this equalizer rule is minimax.

For any estimator $T$ in a metric parameter space, $d(T,f{)}^2+d(T,g{)}^2\ge d(f,g{)}^2/2$ by the triangle inequality. Average the two risk functions and replace their probability density functions by their minimum. This proves $\max (R_f,R_g)\ge d(f,g{)}^2\int \min (p_f,p_g)/4$. The overlap equals one minus total variation distance. The same proof applies to squared error for a real-valued statistical functional of the parameter, without requiring that functional to be injective.

The minimax risk is the infimum over estimators of the supremum of their risk functions over a parameter class. An upper bound supplies an estimator; a lower bound applies to every estimator, often using the Le Cam two-point lemma.

A loss function $L(\theta ,a)$ measures the cost of taking action $a$ at parameter $\theta$. A decision rule $\delta$ selects an action from the data; its risk function is $R(\theta ,\delta )=E_{\theta }L(\theta ,\delta (X))$. A Bayes estimator minimizes the prior-averaged risk, or equivalently minimizes posterior expected loss at almost every observation. An admissible rule has no competing rule of no greater risk everywhere and strictly smaller risk somewhere. To construct a Bayes rule, form the posterior and minimize its expected loss over the allowed actions. Under squared error and finite posterior second moments the unique action is the posterior mean.

Put $S=|X{|}^2$. The given fourth moment and independence give $E_{v}S=nv$, $E_v{S}^2=n(n+2)v^2$. ThusEvery $\alpha \ne 1/(n+2)$ is strictly dominated at every $v>0$ by that coefficient. Hence it cannot be a finite-risk Bayes rule for any proper prior.

To make the second requested comparison, let $m_j=E_{\pi }{v}^j$ and assume finite quadratic Bayes risk. For the affine rules the integrated risk isWith $\alpha$ fixed, the minimum occurs at $\beta =(1-n\alpha )m_1$. Since $m_1>0$, this strictly improves the homogeneous rule unless $\alpha =1/n$, reducing risk by $(1-n\alpha {)}^2{m}_1^2$. Therefore a homogeneous rule with $\alpha \ne 1/n$ is not Bayes, as requested. Combining the two restrictions excludes every homogeneous coefficient.

For a proper prior without global finite moments, the same nonvacuous conclusion follows from the posterior-mean condition. LetFor $s>0$, $M_1^{\prime }(s)=-m(s)/2$. If the posterior mean were $\alpha s$, positivity requires $\alpha >0$ and $M_1=\alpha sm$. Differentiation forces $m(s)=C{s}^{-1-1/(2\alpha )}$. But the marginal density of $S$ would then be a constant times $s^{n/2-2-1/(2\alpha )}$, a pure power not integrable at both zero and infinity. This contradicts a proper prior's normalized predictive distribution. This is the proper-prior obstruction to homogeneous variance Bayes rules.

The finite-loss qualification matters: allowing all posterior losses to equal infinity makes every action a vacuous minimizer. For example, with $n=1$ and proper density $f_0(v)=(1+v{)}^{-2}$, the posterior second moment diverges at positive observations, so a literal infinite-loss definition would defeat the printed assertion. Ordinary meaningful Bayes rules exclude that convention.

A loss function $L(\theta ,a)$ quantifies the cost of taking action $a$ when the parameter is $\theta$. A decision rule $d$ is a measurable map from the observation space to the action space (or, for randomized rules, a conditional action distribution). Its risk function is $R(\theta ,d)={\mathbb{E}}_{\theta }L(\theta ,d(X))$. For a prior probability distribution $\pi$, its Bayes risk isA minimax rule minimizes ${\sup }_{\theta }R(\theta ,d)$.

Put $P=I-p^{-1}{1}_p{1}_p^T$, $Z=PX=X-\bar{X}{1}_p$, $Q=\Vert Z{\Vert }^2$ and $a=p-3$. Componentwise,Thus the James–Stein shrinkage toward the sample mean leaves the average unchanged and multiplies every contrast about it by the same data-dependent factor $1-a/Q$. If $Q>a$ the contrasts shrink without changing sign; if $Q<a$ their signs reverse, and if $Q<a/2$ their magnitudes increase. The event $Q=0$ has probability zero, and the estimator can be assigned $\bar{X}{1}_p$ there.

The sample mean is a linear normal statistic with mean $\bar{\theta }$ and variance $1/p$, soThe error splits into orthogonal mean and contrast components:Since the second term is perpendicular to $1_p$, its cross product with the first is zero for every sample. Taking squared norms and expectations, $p\mathbb{E}(\bar{X}-\bar{\theta }{)}^2=1$, which yields the risk function

To compute the contrast risk, choose an orthonormal basis matrix $U$ for the range of $P$, with $U^{T}U=I_m$, $U{U}^T=P$ and $m=p-1\ge 3$. Then $W=U^{T}X\sim N_m(\eta ,I)$ with $\eta =U^T\theta$, $Q=\Vert W{\Vert }^2$, and the contrast norm is the norm of $(1-a/Q)W-\eta$. Expand it:For $g_j(w)=w_j/\Vert w{\Vert }^2$, the Gaussian Stein identity givesThis application at a singularity is valid: near zero the reciprocal squared radius is integrable in dimension $m>2$, since its radial integral is ${\int }_0^1{r}^{m-3}dr$. Apply Gaussian integration by parts outside a ball of radius $\epsilon$; the inner boundary contribution is bounded by a constant times ${\epsilon }^{m-2}$ and vanishes. The Gaussian tail eliminates the boundary at infinity. Therefore all required expectations are finite and the displayed identity holds.

Since $a=m-2=p-3$, the Stein risk identity now gives contrast risk $m-a^2\mathbb{E}(1/Q)$. Adding the unit mean risk proves

Finally $Q$ has the noncentral chi-squared distribution ${\chi }_m^2(\lambda )$ with $\lambda =\Vert P\theta {\Vert }^2$. To establish where its reciprocal expectation is largest, complete the square in the normal representation $Q=\Vert W{\Vert }^2$ to obtain its Laplace transform:Since $1/q={\int }_0^{\infty }{e}^{-tq}dt$, the Tonelli theorem gives the reciprocal moment of a noncentral chi-squared variableIts integrand is positive and strictly decreasing in $\lambda$ for every $t>0$. The expectation is therefore strictly largest at $\lambda =0$, whereBecause this expectation is subtracted in the risk, the minimizing set is exactly $P\theta =0$, that is, all coordinates of $\theta$ equal. On this line the risk is $p-(p-3)=3$:

For a decision rule $\delta$ and loss $L$, its risk function is $R(\theta ,\delta )={\mathbb{E}}_{\theta }L(\theta ,\delta (X))$. It is an admissible estimator if there is no rule with risk everywhere no larger and somewhere strictly smaller. A Bayes decision rule minimizes $\int R(\theta ,\delta )\pi (d\theta )$ for a specified prior $\pi$. A minimax decision rule minimizes ${\sup }_{\theta }R(\theta ,\delta )$ over all rules.

Here the maximum-likelihood estimator is ${\delta }_0(X)=X$, and its squared-error risk isWrite the proposed estimator as $\delta (X)=X+g(X)$ with $g(x)=-x/\Vert x{\Vert }^2$ for $x\ne 0$, and assign any value at zero, an event of probability zero. Expanding the squared loss givesStein's lemma for a unit-covariance normal vector says $\mathbb{E}[(X_i-{\theta }_i)g_i(X)]=\mathbb{E}[{\partial }_i{g}_i(X)]$ when the relevant derivatives and expectations are integrable. With $r=\Vert x\Vert$,The singularity causes no difficulty in three dimensions: $r^{-2}$ is locally integrable against the volume element $r^{2}drd\Omega$, and the normal tails control infinity. More formally apply Stein's lemma first to $g_{ϵ}(x)=-x/(r^2+ϵ)$, then let $ϵ\downarrow 0$; its squared norm and derivative magnitudes are bounded by constants times $r^{-2}$, so dominated convergence justifies the limit. ConsequentlyThe expectation is finite and strictly positive. Thus this James–Stein estimator strictly dominates the maximum likelihood estimator, which is inadmissible under the specified squared-error loss.

Fix any estimator $\hat{\eta }$ and put $A(x)=d(\hat{\eta }(x),f)$, $B(x)=d(\hat{\eta }(x),g)$ and $D=d(f,g)$. The triangle inequality for the metric gives $A+B\ge D$. HenceThe larger of the two risk functions dominates their average. Using nonnegativity to replace both probability density functions by their minimum,The right side is independent of the estimator, so taking the infimum provesThis metric squared-loss two-point bound uses overlap of the observation laws to quantify how hard it is to distinguish the two separated parameters.

There is a genuine qualification: the first printed inequality is valid as a useful general assertion for $0<\eta <1$, but is false for arbitrary $\eta >0$. Also the displayed likelihood ratio presupposes $P_{f_1}\ll P_{f_0}$. We first prove the intended bound and then give a counterexample to the unrestricted quantifier.

For any estimator $T$, set $A=\{d(T,f_0)<r_n\}$. By the triangle inequality, $d(T,f_1)\ge r_n$ on $A$, while $d(T,f_0)\ge r_n$ on $A^c$. HenceLet $C=\{|Z-1|\le \eta \}$ with $0<\eta <1$. On $C$, $Z\ge 1-\eta$. ThereforeThe last step is Markov inequality. Taking the supremum over the model and then the infimum over estimators proves the intended metric two-point risk bound. At $\eta =1$ the right side is zero. For $\eta >1$ a negative right side is also harmless, but a positive right side need not be a lower bound.

Here is an explicit counterexample with positive probability density functions and $n=3$. Let $A_0=[0,1/2)$, $\delta =1/100$, and take the model consisting ofUse the L2 norm metric, for which $D=d(f_0,f_1)=2(1-2\delta )$, and take $r_n=D/2$ for every $n$. The majority-half decision selects $f_0$ if at least two observations lie in $A_0$. Each error probability is $p=3{\delta }^2-2{\delta }^3=0.000298$, so its maximum normalized risk function is $2p=0.000596$. This is also the likelihood-ratio test, giving ${\mathbb{E}}_{f_0}|Z-1|=2(1-2p)$. At $\eta =3/2$ the proposed right side is $1/12-2p/3>0.083$, contradicting the risk of this explicit estimator. Thus the intended range restriction cannot be omitted.

For the second request use the intended L2 norm metric $d(f,g)=({\int }_0^1(f-g{)}^2{)}^{1/2}$. The PDF defines its square, not the square itself as a metric. PutBoth probability density functions belong to the model, and $d(f_0,f_{1,n})=a_n=2r_n$. The two-point choice may depend on $n$, because the lower bound is applied separately at each sample size. Under $f_0$, the product likelihood ratio $Z_n={\prod }_i(1+a_{n}g(X_i))$ satisfiesThis is the product chi-squared divergence. By Cauchy-Schwarz inequality, ${\mathbb{E}}_0|Z_n-1|<1/4$. With $\eta =1/2$, the proved metric two-point risk bound has normalized right side greater than $1/8$. ConsequentlyThe proof works for every $n\ge 1$ despite the introductory $n>2$. Measurability of the functions defining the probability density functions is understood. This establishes the requested lower bound; it does not assert a matching upper bound over this unrestricted class.

For fixed-design nonparametric regression, let $x_i=i/n$ and $K_h(u)=h^{-1}K(u/h)$. The degree-$p$ local polynomial estimator takes the fitted intercept ${\hat{m}}_h(x;p)={\hat{\beta }}_0$, whereUse a nonnegative regression kernel and an invertible local polynomial Gram matrix to obtain a unique fit. For $p=0$, solving the single weighted least squares normal equation gives the local constant estimator:At the interior point $x_0$, use the standard unit-integral regression kernel convention ${\mu }_0(K)=1$. Otherwise the leading variance below is $R(K)/(nh{\mu }_0(K{)}^2)$; unlike a kernel density estimator, the local constant estimator itself is unchanged by rescaling $K$.

The local constant estimator is a linear estimator in nonparametric regression with weights $W_i=K_h(x_i-x_0)/(n{s}_{0,h})$. The independent unit-variance errors implyTo control the bias of an estimator, differentiability at this one point is enough. WriteSymmetry makes ${\mu }_1(K)=0$, and the supplied moment approximation gives $s_{1,h}=o(h)$. Therefore the linear contribution to the bias of an estimator is $m^{\prime }(x_0)s_{1,h}/s_{0,h}=o(h)$. The remaining contribution has absolute value at most $\eta (h)h$, because only $|x_i-x_0|\le h$ receive positive weight. This is the interior first-order bias cancellation for local constant regression, and gives squared bias of an estimator $o(h^2)$. Combining with the variance provesFor fixed $\epsilon >0$, take $h_{\epsilon }={\epsilon }^{-1/3}{n}^{-1/3}$. It satisfies both asymptotic smoothing bandwidth conditions, and the displayed remainder, multiplied by $n^{2/3}$, tends to zero for this fixed $\epsilon$. ThusLetting $\epsilon \downarrow 0$ proves the optimized scaled pointwise error tends to zero. The order of these limits matters: no remainder uniform in $\epsilon$ has been assumed.

For the lower bound over the whole function class, the absence of a uniform bound on derivatives permits a particularly direct argument. It also avoids assuming a normal distribution for errors when the model only supplies their first two moments. Fix $n$ and $x_0$. Choose a smooth bump function $b$ on the interval, with $b(x_0)=1$, whose support contains no design point except possibly $x_0$. Such a bump exists because the design is finite; at an endpoint use the restriction of a smooth bump on the real line. Every $m_a=ab$, $a\in \mathbb{R}$, belongs to the differentiability class.

If $x_0$ is not a design point, all observation means are zero for every $a$. The Le Cam two-point lemma applied to $a=\pm A$ has total variation distance zero and gives a lower bound tending to infinity with $A$. In fact, the supremum of the mean squared error is infinite at every such $x_0$.

If $x_0$ is a design point, only its response depends on $a$. The resulting location family is the single observation $Y=a+ϵ$; all remaining observations are independent of $a$ and give no extra information. The single-observation location minimax bound isHere is a proof covering discrete as well as continuous errors. Give $a$ the prior distribution $U[-A,A]$, independently of the noise. Put $p_M=\mathbb{P}(|ϵ|\le M)$ and $v_M=\mathrm{Var}(ϵ\mid |ϵ|\le M)$. Consider an easier experiment in which an oracle also reveals whether $|ϵ|\le M$, and reveals $a$ itself on the complementary event. On the event $|ϵ|\le M$ and $|Y|\le A-M$, every possible truncated noise value places $a=Y-ϵ$ within $[-A,A]$. The flat prior distribution then makes the Bayesian posterior noise law exactly its truncated original law. The smallest conditional squared-error loss is its conditional variance $v_M$, attained by the posterior mean.

The probability of that event is at least $p_M(1-2M/A)$ for $A>2M$, by restricting further to $|a|\le A-2M$. Thus every rule's supremum risk function is at least the original Bayes risk, which is at least the easier experiment's Bayes risk, and therefore at least $p_M(1-2M/A)v_M$. Let $A\to \infty$ and then $M\to \infty$. Finite second moment and zero mean give $p_M{v}_M\to 1$. The rule $T(Y)=Y$ has constant mean squared error one, proving the equality.

Consequently, at every $x_0\in [0,1]$, the unrestricted differentiability class satisfies the stronger conclusionThus $c=1$ works for every $n\ge 1$. There is no conflict with the preceding fixed-function limit: the worst functions, including arbitrarily narrow bumps, can vary with $n$. This is the pointwise versus uniform risk distinction. A smoothness ball with a common derivative bound would pose a different minimax risk problem.

Put $s=\sqrt{n}/2$. The posterior is $\mathrm{Beta}(X+s,n-X+s)$, andThe denominator is positive for $n\ge 1$. For $n>4$, both parameters exceed one and the mode simplifies to $(X+s-1)/(n+\sqrt{n}-2)$. For $n=4$ this is the uniform-prior mode, exactly $X/n$; for $n=1,2$ the endpoint or symmetric cases also make the mode rule coincide with $X/n$. Otherwise it is generally a different rule. The posterior mean agrees with $X/n$ only at $X=n/2$.

To find the minimax estimator under quadratic loss, consider the posterior mean ${\delta }_s=(X+s)/(n+2s)$. Its risk function isFor $s^2=n/4$, the numerator is identically $n/4$, soThis constant-risk binomial proportion estimator is the unique Bayes estimator for its proper beta prior under quadratic loss. A competing rule's maximum risk is at least its risk averaged over that prior, which is at least this Bayes risk. The constant-risk rule attains the bound, so the posterior mean in (iii) is the unique minimax rule. Consequently any coincident rule is minimax too: the mean in (i) is the same rule when $n=4$, and the mean in (ii) is the same rule when $n=1$. No posterior mode listed here equals that mean rule for every observation, since at $X=0$ its value is positive whereas their endpoint modes differ from it. This accounts for the small-$n$ coincidences as well as the general answer.

for an estimator $\delta (X)$ and loss $L$, its risk function is $R(p,\delta )={\mathbb{E}}_{p}L(\delta (X),p)$, where expectation is over repeated data at fixed $p$. Its Bayes risk under prior $\pi$ is $r(\pi ,\delta )=\int R(p,\delta )\pi (dp)$, equivalently the joint expectation under the prior and sampling model.

Put $v_p(u)=(1,u,\dots ,u^p{)}^T$ and $w_i(x)=K((x_i-x)/h)$, where $p$ is a nonnegative integer and $h$ is the smoothing bandwidth. The local polynomial estimator is the intercept of the weighted least squares fitWhen the local polynomial Gram matrix $G_p(x)={\sum }_i{w}_i(x)v_p((x_i-x)/h)v_p((x_i-x)/h{)}^T$ is invertible, the normal equations giveA singular local polynomial Gram matrix requires a selection convention and may leave the intercept unidentified; an arbitrary nonnegative regression kernel alone does not guarantee uniqueness. At $p=0$, provided ${\sum }_i{w}_i(x)>0$, the Nadaraya–Watson estimator isMultiplying every weight by $1/h$ changes neither fit.

For the uniform kernel, let $I_x=\{i:|i/n-x|\le h\}$ and $N_x=|I_x|$. Then ${\hat{m}}_h(x)=N_x^{-1}{\sum }_{i\in I_x}{Y}_i$. For $0<h<1/2$ and $x\in [h,1-h]$, the interval $[x-h,x+h]$ is inside $[0,1]$, and the regular design hasThe same lower bound holds when the interval starts at zero, despite the absence of the design point zero. In particular the Nadaraya–Watson estimator is defined throughout the integration interval.

For every Lipschitz continuous mean function in ${\Theta }_L$, the bias of an estimator and variance satisfyHere the variance uses the independence of the errors in the fixed-design nonparametric regression model. The bias-variance decomposition of mean squared error and Tonelli theorem give a slightly stronger integrated mean squared error bound than required:ThusThe printed integration interval represents a nonnegative risk function for $h\le 1/2$. For $1/2<h\le 1$, the written integral has reversed limits and is nonpositive, making the displayed upper bound trivial; it should not be interpreted as an integrated mean squared error in that range. For $h>1$, evaluation outside $[0,1]$ would additionally require a specified extension of the mean function. The subsequent optimization restricts to $h<1/3$.

Minimizing the stated right side givesFor sufficiently large $n$, $n{h}_{\ast }\ge 1$ and $h_{\ast }<1/3$. For example take $n_0=1+\lceil \max \{1,2L/(\sqrt{3}\sigma ),81{\sigma }^2/(4L^2)\}\rceil$. Therefore the upper bound holds with

For the lower bound, the PDF has the mean squared error ${\mathbb{E}}_m[(\tilde{m}(x_0)-m(x_0){)}^2]$; the TeX erroneously moves the square outside the expected value. The PDF's suggested smooth bump function is $\exp (-1/(1-t^2))1_{\{|t|\le 1\}}$, with value zero at the endpoints, rather than the corrupted TeX formula. A triangular alternative suffices because only Lipschitz continuity is needed.

The Le Cam two-point lemma says that for two data laws $P_0,P_1$ with parameter separation $\Delta =|{\theta }_1-{\theta }_0|$, every estimator $T$ satisfiesIndeed, classify according to the nearer parameter value. Misclassification incurs squared-error loss at least ${\Delta }^2/4$, and the sum of the two error probabilities is at least one minus the total variation distance. Averaging the two risk functions proves the stated form.

Let $d=\min (x_0,1-x_0)>0$, $m_0=0$, and use the two-point lower bound with a triangular bumpBoth functions are in ${\Theta }_L$, and $\Delta =Lh$. Assume first that $h\le d$ and $nh\ge 1$. At most $2nh+1$ design points meet the bump's support. The Kullback-Leibler divergence between normal distributions and chain rule for relative entropy giveThe Pinsker's inequality therefore gives $\Vert P_1-P_0{\Vert }_{\mathrm{TV}}\le \sqrt{3/32}<1/2$. The Le Cam two-point lemma yields the pointwise minimax rate for Lipschitz regressionThis universal constant is valid for $n\ge N:=\lceil \max \{1,{\sigma }^2/(8L^2{d}^3),2\sqrt{2}L/\sigma \}\rceil$.

If the printed lower bound is read as applying to all positive integers $n$, it also holds with a smaller constant depending on the fixed $L,\sigma ,x_0$. For $n<N$, use the constant mean functions $m_0=0$ and $m_1=\sigma /\sqrt{n}$. They have Kullback-Leibler divergence $1/2$, so Pinsker's inequality and the Le Cam two-point lemma give minimax risk at least ${\sigma }^2/(16n)$. Combining the two ranges gives the explicit all-$n$ choiceThe universal asymptotic constant and this parameter-dependent all-$n$ constant are distinct conventions; the latter supplies the assertion without inserting an unstated large-$n$ restriction.

The risk function of an estimator $\hat{\theta }$ under the stated quadratic loss isThe multivariate normal density is proportional, as a function of $\theta$, toso maximum likelihood estimation gives ${\hat{\theta }}_{\mathrm{MLE}}=X$. Writing $X=\theta +Z$ with $Z\sim N_p(0,I_p)$,for every $\theta \in {\mathbb{R}}^p$.

Since ${\overline{X}}_n\sim N(\theta ,1/n)$, the estimator has variance
$a^2/n$ and bias $(a-1)\theta +b$. Its quadratic risk function is

A risk function can decay rapidly for each fixed regression function while its supremum over a function class decays more slowly or fails to decay. Bounds depending on each function's local differentiability remainder need not be uniform. In fixed-design nonparametric regression, a class with no common smoothness bound permits smooth bump functions whose supports shrink between design points. Such sequences obstruct uniform estimation without contradicting fixed-function mean squared error bounds.