Le Cam two-point lemma

Le Cam's two-point lemma lower-bounds minimax risk by reducing estimation to testing two separated parameter values. Under squared loss, one form is

If two parameters are separated by at least $2r$ in a metric, any estimator gives a test by deciding whether its distance from the first parameter is below $r$. For $0<\eta <1$, the event $|Z-1|\le \eta$ and Markov inequality give the displayed minimax risk bound, where $Z$ is the product likelihood ratio. The unrestricted claim for every positive $\eta$ is false when both factors on the right are negative.

For any estimator $T$ in a metric parameter space, $d(T,f{)}^2+d(T,g{)}^2\ge d(f,g{)}^2/2$ by the triangle inequality. Average the two risk functions and replace their probability density functions by their minimum. This proves $\max (R_f,R_g)\ge d(f,g{)}^2\int \min (p_f,p_g)/4$. The overlap equals one minus total variation distance. The same proof applies to squared error for a real-valued statistical functional of the parameter, without requiring that functional to be injective.

For estimating the expected value over all continuous probability density functions on $[0,1]$, compare $f=1$ and $g_n(u)=1+(u-1/2)/\sqrt{n}$. Their means differ by $1/(12\sqrt{n})$, and ${\chi }^2(P_{g_n}\Vert P_f)=1/(12n)$. The chi-squared divergence of product measures bounds joint total variation distance by $1/2$. The metric squared-loss two-point bound then gives the displayed uniform minimax risk lower bound.

The Cauchy-Schwarz inequality gives $\Vert P-Q{\Vert }_{\mathrm{TV}}=\frac{1}{2}{\mathbb{E}}_Q|dP/dQ-1|\le \frac{1}{2}\sqrt{{\chi }^2(P\Vert Q)}$. Every statistical hypothesis testing rule has sum of its Type I error and Type II error at least $1-\Vert P-Q{\Vert }_{\mathrm{TV}}$, proving the displayed bound. A mixture model of alternatives yields a lower bound on the worst alternative error because a maximum dominates an average.

For two observation laws $P_0,P_1$ at real parameters separated by $\Delta$, every estimator $T$ satisfies ${\max }_j{\mathbb{E}}_j|T-{\theta }_j|\ge \Delta (1-\mathrm{TV}(P_0,P_1))/2$. With common densities, the sum of the two risks is at least $\int \min (p_0,p_1)(|T-{\theta }_0|+|T-{\theta }_1|)$, at least $\Delta \int \min (p_0,p_1)=\Delta (1-\mathrm{TV})$ by the triangle inequality. Divide by two. The same proof works for any metric loss.

For $n$ independent observations with a normal distribution $N(\mu ,{\sigma }^2)$, $\sigma >0$ fixed, every estimator has ${\sup }_{\mu }{\mathbb{E}}_{\mu }|T-\mu |\ge \sigma /(4\sqrt{2n})$. Compare ${\mu }_0=0$ and ${\mu }_1=\sigma /\sqrt{2n}$: the joint Kullback-Leibler divergence is $1/4$, so the total variation–Hellinger–relative entropy inequality bounds total variation distance by $1/2$. Apply the Le Cam lower bound under absolute-error loss. Degenerate zero-noise laws do not satisfy a positive lower bound.

Consider fixed-design nonparametric regression $Y_i=m(i/n)+{\epsilon }_i$, with independent errors ${\epsilon }_i\sim N(0,{\sigma }^2)$, and estimation at a fixed interior point $x_0\in (0,1)$. In the class of functions with Lipschitz continuity constant at most $L$, compare $m_0=0$ with $m_1(x)=L(h-|x-x_0|{)}_{+}$, with $h$ small enough to stay inside the design interval.

The target values differ by $Lh$. At most $2nh+1$ equally spaced design points meet the bump, and its height is at most $Lh$, so the joint Kullback-Leibler divergence is bounded byTaking $h$ a sufficiently small constant times $({\sigma }^2/(n{L}^2){)}^{1/3}$ keeps this bounded away from a perfectly distinguishable experiment. Pinsker's inequality and the Le Cam two-point lemma give squared-error risk at least a constant times $L^{2/3}{\sigma }^{4/3}{n}^{-2/3}$. The design-spacing assumption is essential to the count of observations inside the bump; it is not valid for arbitrary fixed designs.