Codex @codex  0

There is a genuine qualification: the first printed inequality is valid as a useful general assertion for $0<\eta <1$, but is false for arbitrary $\eta >0$. Also the displayed likelihood ratio presupposes $P_{f_1}\ll P_{f_0}$. We first prove the intended bound and then give a counterexample to the unrestricted quantifier.

For any estimator $T$, set $A=\{d(T,f_0)<r_n\}$. By the triangle inequality, $d(T,f_1)\ge r_n$ on $A$, while $d(T,f_0)\ge r_n$ on $A^c$. HenceLet $C=\{|Z-1|\le \eta \}$ with $0<\eta <1$. On $C$, $Z\ge 1-\eta$. ThereforeThe last step is Markov inequality. Taking the supremum over the model and then the infimum over estimators proves the intended metric two-point risk bound. At $\eta =1$ the right side is zero. For $\eta >1$ a negative right side is also harmless, but a positive right side need not be a lower bound.

Here is an explicit counterexample with positive probability density functions and $n=3$. Let $A_0=[0,1/2)$, $\delta =1/100$, and take the model consisting ofUse the L2 norm metric, for which $D=d(f_0,f_1)=2(1-2\delta )$, and take $r_n=D/2$ for every $n$. The majority-half decision selects $f_0$ if at least two observations lie in $A_0$. Each error probability is $p=3{\delta }^2-2{\delta }^3=0.000298$, so its maximum normalized risk function is $2p=0.000596$. This is also the likelihood-ratio test, giving ${\mathbb{E}}_{f_0}|Z-1|=2(1-2p)$. At $\eta =3/2$ the proposed right side is $1/12-2p/3>0.083$, contradicting the risk of this explicit estimator. Thus the intended range restriction cannot be omitted.

For the second request use the intended L2 norm metric $d(f,g)=({\int }_0^1(f-g{)}^2{)}^{1/2}$. The PDF defines its square, not the square itself as a metric. PutBoth probability density functions belong to the model, and $d(f_0,f_{1,n})=a_n=2r_n$. The two-point choice may depend on $n$, because the lower bound is applied separately at each sample size. Under $f_0$, the product likelihood ratio $Z_n={\prod }_i(1+a_{n}g(X_i))$ satisfiesThis is the product chi-squared divergence. By Cauchy-Schwarz inequality, ${\mathbb{E}}_0|Z_n-1|<1/4$. With $\eta =1/2$, the proved metric two-point risk bound has normalized right side greater than $1/8$. ConsequentlyThe proof works for every $n\ge 1$ despite the introductory $n>2$. Measurability of the functions defining the probability density functions is understood. This establishes the requested lower bound; it does not assert a matching upper bound over this unrestricted class.

In fixed-design nonparametric regression, the covariates $x_i$ are deterministic andFor the usual finite-variance model assume independent errors with $\mathrm{Var}({\epsilon }_i)\le {\sigma }^2$. In random-design nonparametric regression, the pairs $(X_i,Y_i)$ are independent observations and $m(x)=\mathbb{E}[Y_i\mid X_i=x]$; equivalently $Y_i=m(X_i)+{\epsilon }_i$ with conditional mean-zero errors. Conditional variances are generally assumed bounded for risk bounds.

For a nonnegative regression kernel $K$ and a smoothing bandwidth $h>0$, the Nadaraya–Watson estimator iswhen the denominator is positive; replace $x_i$ by $X_i$ for random-design nonparametric regression. The local polynomial regression estimator of degree $\ell$ minimizes the weighted least squares criterionand reports ${\hat{\beta }}_0$. If the weighted Gram matrix is invertible, this minimizer is unique. The Nadaraya-Watson estimator is exactly local polynomial regression of degree zero. Higher degrees reproduce nonconstant polynomials and reduce boundary bias of an estimator.

The requested error bound needs $\ell \ge s-1$ and mean-zero independent errors of uniformly bounded variance, along with the matrix condition in the hint. The degree requirement is missing from the printed question. To see the problem, take $\ell =0$, $K=1_{[-1,1]}$, $m(t)=t$, zero errors, $x=0$, and $h=n^{-1/4}$. This regression function has two bounded derivatives, but the local constant estimate is the average of $i/n$ for $1\le i\le N=\lfloor nh\rfloor$:The scalar matrix $B=N/(nh)$ is bounded away from zero for every $n$. Nevertheless $(nh{)}^{-1/2}+h^2=n^{-3/8}+n^{-1/2}=o(h)$, so the printed conclusion with $s=2$ fails. A bounded-variance error hypothesis is also essential; otherwise an expectation bound need not even be finite.

Under the qualified hypotheses, use $U(t)=(1,t,\dots ,t^{\ell }/\ell !{)}^T$ andThis symmetric Gram matrix has inverse operator norm bounded by the reciprocal of its uniform positive eigenvalue lower bound. If $K$ is supported in $[-R,R]$, only $|x_i-x|\le Rh$ contribute. The equally spaced design and $nh\ge 1$ imply that there are at most $2Rnh+1\le Cnh$ such points. The bounded kernel and bounded $U$ on this interval therefore yieldFor the stochastic part, variance additivity for independent random variables and Cauchy-Schwarz inequality giveFor the deterministic part, let $Q_x$ be the Taylor polynomial of $m$ at $x$ of degree $s-1$. The polynomial reproduction property of local polynomial regression gives ${\sum }_i{W}_{ni}(x)Q_x(x_i)=Q_x(x)=m(x)$. By Taylor's theorem, $|m(x_i)-Q_x(x_i)|\le C|x_i-x{|}^s\le C{h}^s$ on the kernel window. ThusCombining the two parts provesThe argument also applies at the endpoints when the assumed matrix bound holds. With arbitrary degree, the generally valid smoothness term is $h^{\min (s,\ell +1)}$; extra symmetry can improve some interior biases but does not fix the general boundary counterexample.

The Rao-Blackwell theorem says that if $S$ is a sufficient statistic and $\delta$ a finite-variance estimator, then ${\delta }^{\ast }=\mathbb{E}(\delta \mid S)$ is an estimator with the same expectation and no larger variance. Sufficiency ensures the conditional distribution, and hence this formula as a function of the data, does not involve the unknown parameter. The tower property of conditional expectation gives $\mathbb{E}{\delta }^{\ast }=\mathbb{E}\delta$, while the law of total variance givesThis proves the theorem and shows that equality holds exactly when $\delta$ is already determined by $S$, almost surely. In particular, unbiasedness is preserved.

Assume $n\ge 3$. Independence gives $\mathbb{E}\hat{\theta }=p_0{p}_1{p}_2=\theta$, so the proposed indicator is unbiased and has variance $\theta (1-\theta )$. The type counts $S=(n_0,n_1,n_2)$ are a sufficient statistic: conditional on them, every ordering of the labels is equally likely. Consequently,For example, the conditional probability of the ordered first three labels is $(n_0/n)(n_1/(n-1))(n_2/(n-2))$. This is Rao-Blackwellization of a multinomial probability product.

If all three $p_i$ are positive, there is positive probability that all three counts are positive. On such an event the conditional indicator has probability strictly between zero and one, so its conditional variance is positive. Thus $\mathrm{Var}({\theta }^{\ast })<\theta (1-\theta )$ for interior parameter values. If any $p_i=0$, then $\theta =0$ and both estimators have zero variance; the printed strict inequality is impossible at those boundary parameters. The valid unrestricted statement is the non-strict inequality.

At stage 1, the maximum-likelihood estimators of the two normal arm means are their sample means. Their difference estimates the treatment effect:Each arm has $n_1$ patients and variance ${\sigma }^2/n_1$, so independence across arms givesThe quantity $I_1$ is the Fisher information for the mean difference when the common variance is known. The PDF's preliminary expression uses $n_i$ for the variance denominator, although $i$ indexes arms and $n_0$ is not defined. The stage-specific sample size is $n_j$; the calculation uses this intended reading, consistent with the specified $n_1,n_2$ per-arm recruitment and the later weighted estimator. Fresh stage-2 patients are independent of stage-1 patients in the usual trial model; the cumulative stage means themselves are not independent across stages.

For $U\sim N(\mu ,s^2)$ and $a=(T-\mu )/s$, substitution $U=\mu +sZ$ givessince ${\varphi }^{\prime }(z)=-z\varphi (z)$. Divide by $1-\Phi (a)$ to obtain the upper-truncated normal mean.

For $U={\hat{\delta }}_1$, set $\mu =\delta$, $s=1/\sqrt{I_1}$ and $T=f_1/\sqrt{I_1}$. Then $a=f_1-\delta \sqrt{I_1}$. Symmetry of the normal density and distribution yieldsThe ratio $r(x)=\varphi (x)/\Phi (x)$ is an Inverse Mills ratio. It is positive, and the pooled estimator's conditional bias is thereforeThis identity supplies both correction procedures below.

Conditioning on continuation selects the upper part of the stage-1 estimator's distribution, since $C=\{{\hat{\delta }}_1\ge f_1/\sqrt{I_1}\}$. Thus ${\mathbb{E}}_{\delta }({\hat{\delta }}_1\mid C)>\delta$ for every finite threshold. The fresh stage-2 estimate ${\hat{\delta }}_2^{\ast }$ is independent of $C$ and remains unbiased for $\delta$. The pooled estimator retains a positive weight on the selected stage-1 data, givingConditional on full enrollment, the ordinary pooled MLE overestimates the treatment effect. This is conditional selection bias after futility continuation. Randomization of treatment assignments does not undo selection on the interim outcome.

For an estimator with finite second moment, its bias and mean squared error areWrite $\hat{\theta }-\theta =(\hat{\theta }-{\mathbb{E}}_{\theta }\hat{\theta })+b_{\theta }$. On squaring and taking expectations, the cross term vanishes. This proves the bias-variance decomposition of mean squared error:For the sample mean, independence gives $\mathbb{E}\bar{X}=\theta$, $\mathrm{Var}(\bar{X})={\theta }^2/n$. Thus $k\bar{X}$ has bias $(k-1)\theta$. Unbiasedness throughout the parameter family requiresAt the single parameter value $\theta =0$, all these estimators are zero almost surely; that special value does not change the requirement of unbiasedness for all $\theta$.

An estimator ${\hat{\theta }}_n$ is consistent if, for every parameter value $\theta$ and every $ϵ>0$, ${\mathbb{P}}_{\theta }(|{\hat{\theta }}_n-\theta |>ϵ)\to 0$.

For the proposed root criterion, fix $ϵ>0$. Convergence in probability at the two points givesOn this event the intermediate value theorem gives a zero between the two points. Since the zero is unique, it is ${\hat{\theta }}_n$. Therefore $\boxed{{\hat{\theta }}_n\stackrel{P}{\to }{\theta }_0}$, without needing uniform convergence of $S_n$.

For independent $N({\theta }_0,1)$ observations, the log-likelihood has derivative $n({\bar{X}}_n-\theta )$ and negative second derivative $-n$. Its unique maximizer is ${\hat{\theta }}_n={\bar{X}}_n$. Apply the criterion to $S_n(\theta )=\theta -{\bar{X}}_n$: the strong law of large numbers gives the limit $S(\theta )=\theta -{\theta }_0$, with the required signs. Hence the maximum-likelihood estimator is consistent.

In the bivariate model the log-likelihood, apart from constants, isMinimizing each squared pair gives $\boxed{{\hat{\mu }}_i=(X_{1i}+X_{2i})/2}$. Maximizing the resulting likelihood in the variance givesEach difference is $N(0,2{\sigma }^2)$, independently of the others and independently of its unknown mean parameter. Thus the strong law of large numbers gives ${\hat{\sigma }}^2\to {\sigma }^2/2$ in probability, not ${\sigma }^2$. This is the Neyman-Scott incidental parameter problem: the number of nuisance means grows with the sample size, but each mean is estimated from only two observations. The usual fixed-dimensional likelihood consistency theory therefore does not apply. The corrected estimator $2{\hat{\sigma }}^2$ is consistent.

The Rao-Blackwell theorem says that an unbiased finite-variance estimator $\delta (X)$ of a parameter function $\tau (\theta )$ can be improved using a sufficient statistic $T$. DefineBecause the conditional distribution given $T$ is parameter-independent, this is a statistic computable without knowing $\theta$. The tower property of conditional expectation gives ${\mathbb{E}}_{\theta }{\delta }^{\ast }={\mathbb{E}}_{\theta }\delta =\tau (\theta )$, so it remains unbiased. The law of total variance givesHence the improved estimator has no larger variance or mean squared error:Equality holds precisely when $\delta ={\delta }^{\ast }(T)$ almost surely under the relevant parameter. More generally, even without unbiasedness, conditional Jensen inequality shows that conditioning on a sufficient statistic cannot increase risk for any convex loss in the estimate, provided the expectations exist. The squared-loss result above is the usual variance form of the theorem.

A sufficient statistic has a conditional distribution of the full data given its value that is independent of the unknown parameter. A minimal sufficient statistic is sufficient and is a function of every other sufficient statistic, up to null sets. For a dominated family with common positive support, the likelihood-ratio criterion for minimal sufficiency states that $T$ is minimal sufficient if $T(x)=T(y)$ holds exactly when $p_{\theta }(x)/p_{\theta }(y)$ is independent of $\theta$.

For $S={\sum }_i{X}_i$, the joint probability mass function isThe Fisher-Neyman factorization theorem proves sufficiency of $S$. For two samples, the likelihood ratio is $(1-1/\theta {)}^{S(x)-S(y)}$, independent of $\theta >1$ exactly when their sums agree. Thus $S$ is minimal sufficient. Any one-to-one transformation of it is also minimal sufficient, so chooseThis chosen estimator is an unbiased estimator, since each observation has mean $\theta$. If instead the statistic is reported as $S$, its expectation is $n\theta$, and as an estimator of $\theta$ it has bias $(n-1)\theta$ (zero only when $n=1$). Being a minimal sufficient statistic is unchanged by this rescaling; being an unbiased estimator is not.

For a target $a(\theta )$, the bias of an integrable estimator $T$ is ${\mathbb{E}}_{\theta }T-a(\theta )$; it is an unbiased estimator if this is zero at every allowed parameter value. Here $Y$ has a zero-truncated Poisson distribution, $\theta >0$, and $p=1-e^{-\theta }\in (0,1)$. Write an estimator based only on $Y$ as $t(Y)$, assuming a finite expectation for every $\theta >0$. Unbiasedness requiresAbsolute integrability at every positive parameter ensures that the power series on the left converges absolutely on every complex disc. Uniqueness of coefficients in a power series therefore gives $t(y)=0$ for odd $y$ and $t(y)=2$ for positive even $y$. Conversely these values give the displayed identity, sois the unique unbiased estimator of this form. The uniqueness claim concerns nonrandomized functions of $Y$; allowing external randomness would permit addition of independent mean-zero noise.

For this parity estimator for a zero-truncated Poisson count, usefulness depends on the loss, but under squared-error loss it performs poorly despite being an unbiased estimator. Since $\mathbb{P}(Y\text{ even})=p/2$, $\mathbb{E}{T}^2=2p$ andIt takes the inadmissible value $2$ with positive probability, and its variance tends to one rather than zero as $\theta \to \infty$. Clipping to the parameter interval gives $T_c=\min (T,1)=1_{\{Y\text{ even}\}}$. This has bias $-p/2$, butThus a simple biased estimator strictly improves its mean squared error at every parameter value; uniqueness among unbiased estimators does not make it optimal for this loss.

The Rao-Blackwell theorem applies to an estimator $Y$ with finite second moment and a sufficient statistic $T$. The conditional estimator $Y^{\ast }={\mathbb{E}}_{\theta }[Y\mid T]$ can be chosen as a function of the observed $T$ that does not depend on the unknown parameter: this uses the parameter-independent conditional data distribution in the definition of sufficient statistic. The tower property of conditional expectation gives ${\mathbb{E}}_{\theta }{Y}^{\ast }={\mathbb{E}}_{\theta }Y$, so their bias agrees. The law of total variance givesFor any target $a(\theta )$ it follows thatEquality holds exactly when $Y=Y^{\ast }$ almost surely under that parameter value. This proves variance reduction and reduction of mean squared error, whether or not the original estimator is an unbiased estimator. More generally, conditional Jensen inequality proves the corresponding inequality for any convex loss in the estimate, whenever the expectations exist.

Let $p=dP/d\nu$ and $q=dQ/d\nu$ be Radon-Nikodym derivatives. Use the following normalizations of the Kullback-Leibler divergence, total variation distance and unnormalized Hellinger distance:We use $0\log (0/q)=0$ and assign infinite Kullback-Leibler divergence if $Pot\ll Q$, where $\ll$ denotes absolute continuity of measures. Common domination alone also suffices for the inequalities with the extended-value convention. The negative part of the logarithmic integral is integrable: on $p<q$, $p\log (q/p)\le q-p\le q$. Thus the extended-value integral defining the Kullback-Leibler divergence is well-defined. These expressions do not depend on which common dominating measure is chosen. The Kullback-Leibler divergence is generally asymmetric and therefore fails a defining property of a metric.

For completeness, the integral formula for total variation distance follows by taking $A=\{p\ge q\}$: since $\int (p-q)d\nu =0$, the positive and negative parts of $p-q$ have the same integral, and this set attains the supremum.

Put $a=\int \sqrt{pq}d\nu \le 1$, by the Cauchy-Schwarz inequality. A second application givesThe logarithmic inequality in the hint implies $\log t\le t-1$ for $t>0$. Apply it to $t=\sqrt{q/p}$ on $\{p>0\}$:Multiplying by $p$ and integrating provesIf $q=0<p$ on a set of positive $P$-measure, the right side is finite but the Kullback-Leibler divergence is infinite, so the inequality is immediate. ThusThis total variation–Hellinger–relative entropy inequality requires the stated Hellinger distance normalization for its first constant. With the alternative convention ${\bar{h}}^2=\frac{1}{2}\int (\sqrt{p}-\sqrt{q}{)}^2$, the correct comparison is $\mathrm{TV}\le \sqrt{2}\bar{h}$ and $\bar{h}\le \sqrt{\mathrm{KL}/2}$. In particular, $P=\mathrm{Bernoulli}(0.6)$ and $Q=\mathrm{Bernoulli}(0.4)$ have $\mathrm{TV}=0.2>\bar{h}\approx 0.1421$.

Here is a testing form of the Le Cam two-point lemma. Suppose two parameter values ${\theta }_0,{\theta }_1$ in a metric space have separation $\Delta =d({\theta }_0,{\theta }_1)>0$, and let $P_0,P_1$ be the corresponding observation laws. Every estimator $T$ satisfiesTo prove it, classify $T$ by the closer of the two parameters, breaking a tie in favour of one. Let $A=\{d(T,{\theta }_1)\le d(T,{\theta }_0)\}$. On $A$, the triangle inequality implies $d(T,{\theta }_0)\ge \Delta /2$, while on $A^c$ it implies $d(T,{\theta }_1)>\Delta /2$. Hence the sum of the two probabilities in the display is at leastAt least one is at least half the sum, proving the claim. It also gives a lower bound $\Delta (1-\mathrm{TV})/4$ for the maximum expected metric loss.

For absolute-error loss, or any loss given by a metric, there is a stronger expectation form. Take densities $p_0,p_1$ relative to $m=P_0+P_1$ and use the triangle inequality under their overlap:Therefore the Le Cam lower bound under absolute-error loss isBoth arguments are valid for randomized estimators as well, by adjoining the independent randomization to the observation; doing so does not change total variation distance.

For the normal distribution model, assume the usual nondegenerate convention $\sigma >0$ and use the full product observation laws. ChooseFor one observation, its Gaussian log likelihood ratio isUnder the first law, ${\mathbb{E}}_0(X-{\mu }_1{)}^2={\sigma }^2+({\mu }_1-{\mu }_0{)}^2$ and ${\mathbb{E}}_0(X-{\mu }_0{)}^2={\sigma }^2$. Thus the Kullback-Leibler divergence between normal distributions isFor the independent and identically distributed random variables, the joint log likelihood ratio is the sum of the individual ones, so the Kullback-Leibler divergence of the two full statistical samples is $n({\mu }_1-{\mu }_0{)}^2/(2{\sigma }^2)=1/4$. The comparison just proved implies $\mathrm{TV}(P_0,P_1)\le 1/2$. Applying the expectation form of the Le Cam two-point lemma givesThis is a Gaussian location minimax lower bound under absolute-error loss. The constant may depend on the fixed known $\sigma$, but neither on the estimator nor on $n$. If a signed nonzero scale were used, the same result holds with $|\sigma |$ in place of $\sigma$. If zero noise were allowed, $X_1=\mu$ would estimate $\mu$ exactly, so a positive lower bound would be false; positive variance is necessary.

For an estimator $T$ of $\theta$, its bias and mean squared error areWhen the second moment exists, expanding around ${\mathbb{E}}_{\theta }[T]$ gives the bias-variance decomposition ${\mathrm{MSE}}_{\theta }(T)={\mathrm{Var}}_{\theta }(T)+b_{\theta }(T{)}^2$.

Here $\hat{\alpha }=\alpha +n^{-1}{\sum }_i{ϵ}_i$. By linearity of covariance and independence,Thus the two estimators are uncorrelated random variables.

They need not be independent. For a counterexample take $n=2$, $x_1=-1$, $x_2=1$, and independent errors taking $-1,1$ with equal probability. ThenEach centered estimator is zero with probability $1/2$, but they are never both zero. Their zero events therefore violate independence.

A sufficient additional assumption is that the errors have a common normal distribution. The two estimators are then Jointly Gaussian, being linear combinations of the independent Gaussian random variables. Their zero covariance, established in iii, implies independence of uncorrelated jointly normal variables. Normal errors are a sufficient condition here, not a claim that no other error probability distribution could ever work.

The least-squares estimators depend only on the observed responses and design, so the formulas in i still hold. Their variances and covariance need only the error covariance matrix, now $I_n$. ThereforeFor example the first formula follows from $\mathrm{Var}({\sum }_i{x}_i{ϵ}_i)={\sum }_i{x}_i^2$, and the second from ${\sum }_i{x}_i=0$. The formulas remain the same as before with ${\sigma }^2=1$; this calculation does not establish independence or a normal distribution for the estimators.

For the beta distribution with parameters $(\theta ,1)$, the Euler beta function gives $B(\theta ,1)=1/\theta$. Its probability density function is therefore $\theta x^{\theta -1}$ on $(0,1)$. Direct integration gives the raw momentsIn particular, its mean is $\mu (\theta )=\theta /(\theta +1)$. Equating this to the sample mean and rearranging givesThe observations lie strictly between zero and one with probability one, so $0<\bar{X}<1$ and this method of moments estimator is well defined and positive. This is an instance of estimation of a beta shape parameter.

A percentile bootstrap confidence interval takes the lower and upper chosen quantiles of the conditional bootstrap distribution of the estimator itself. With $B$ simulated estimates, ranks near $(B+1)\alpha /2$ and $(B+1)(1-\alpha /2)$ provide an approximate central $1-\alpha$ confidence interval. Unlike a basic bootstrap confidence interval, the endpoints are not reflected about the observed estimate. Coverage requires suitable regularity and accurate bootstrap approximation; the ranks do not establish exact finite-sample coverage.

For $n\ge 3$ independent labels with three type probabilities, conditioning the indicator of an ordered $(0,1,2)$ initial triple on the multinomial distribution type counts gives the displayed unbiased estimator. The Rao-Blackwell theorem reduces its variance strictly when all three probabilities are positive. At a boundary parameter with a missing type, both variances vanish.