Statistical decision theory

Statistical decision theory compares decision rules through the expected loss they incur under each parameter value.

The loss $|a-\theta |$ for estimating a real parameter $\theta$ by $a$. Its risk of a decision rule is the corresponding expected value. Unlike squared-error loss, its pointwise size is a metric, so the triangle inequality gives a direct overlap version of the Le Cam two-point lemma.

Quadratic loss penalizes an estimate $a$ of a vector parameter $\theta$ by $\Vert a-\theta {\Vert }^2$. Its posterior expected value is minimized by the posterior mean.

The risk of an estimator under squared Euclidean error is its expected squared distance from the true parameter. Pointwise risk domination can be strict even when the supremum risk agrees, because the supremum can be approached at parameters tending to infinity.

The mean-vector prediction risk measures squared Euclidean norm error in estimating the deterministic response mean. For independent future noise with covariance matrix ${\sigma }^2{I}_n$, prediction of a new noisy response adds $n{\sigma }^2$ to this risk. A Gaussian orthogonal projection matrix $P$ of rank $k$ has risk $\Vert (I-P)\mu {\Vert }^2+k{\sigma }^2$.

For $Y\sim N_n(\mu ,{\sigma }^{2}I)$ and a fixed orthogonal projection matrix $P$ of rank $k$, the displayed expression is an unbiased estimator of the mean-vector prediction risk whenever ${\hat{\sigma }}^2$ is an unbiased estimator of ${\sigma }^2$. Independence of its two terms is unnecessary. Comparing fixed models yields the Mallows Cp penalty, but minimizing unbiased estimates does not preserve unbiasedness after selection.

For a normal distribution $N_n(\mu ,{\sigma }^{2}I)$ with unrestricted $\mu \in {\mathbb{R}}^n$ and unknown ${\sigma }^2$, an integrable data-only unbiased estimator of the mean-vector prediction risk of a fixed rank-$k$ orthogonal projection matrix exists exactly when $2k=n$. To prove necessity, randomize the mean by independent Gaussian variance $v$: conditioning adds $(n-k)v$ to the squared bias term, whereas the marginal variance identity would add $kv$. When $2k=n$, the residual sum of squares itself is unbiased.

Le Cam's two-point lemma lower-bounds minimax risk by reducing estimation to testing two separated parameter values. Under squared loss, one form is

If two parameters are separated by at least $2r$ in a metric, any estimator gives a test by deciding whether its distance from the first parameter is below $r$. For $0<\eta <1$, the event $|Z-1|\le \eta$ and Markov inequality give the displayed minimax risk bound, where $Z$ is the product likelihood ratio. The unrestricted claim for every positive $\eta$ is false when both factors on the right are negative.

For any estimator $T$ in a metric parameter space, $d(T,f{)}^2+d(T,g{)}^2\ge d(f,g{)}^2/2$ by the triangle inequality. Average the two risk functions and replace their probability density functions by their minimum. This proves $\max (R_f,R_g)\ge d(f,g{)}^2\int \min (p_f,p_g)/4$. The overlap equals one minus total variation distance. The same proof applies to squared error for a real-valued statistical functional of the parameter, without requiring that functional to be injective.

For estimating the expected value over all continuous probability density functions on $[0,1]$, compare $f=1$ and $g_n(u)=1+(u-1/2)/\sqrt{n}$. Their means differ by $1/(12\sqrt{n})$, and ${\chi }^2(P_{g_n}\Vert P_f)=1/(12n)$. The chi-squared divergence of product measures bounds joint total variation distance by $1/2$. The metric squared-loss two-point bound then gives the displayed uniform minimax risk lower bound.

The Cauchy-Schwarz inequality gives $\Vert P-Q{\Vert }_{\mathrm{TV}}=\frac{1}{2}{\mathbb{E}}_Q|dP/dQ-1|\le \frac{1}{2}\sqrt{{\chi }^2(P\Vert Q)}$. Every statistical hypothesis testing rule has sum of its Type I error and Type II error at least $1-\Vert P-Q{\Vert }_{\mathrm{TV}}$, proving the displayed bound. A mixture model of alternatives yields a lower bound on the worst alternative error because a maximum dominates an average.

For two observation laws $P_0,P_1$ at real parameters separated by $\Delta$, every estimator $T$ satisfies ${\max }_j{\mathbb{E}}_j|T-{\theta }_j|\ge \Delta (1-\mathrm{TV}(P_0,P_1))/2$. With common densities, the sum of the two risks is at least $\int \min (p_0,p_1)(|T-{\theta }_0|+|T-{\theta }_1|)$, at least $\Delta \int \min (p_0,p_1)=\Delta (1-\mathrm{TV})$ by the triangle inequality. Divide by two. The same proof works for any metric loss.

For $n$ independent observations with a normal distribution $N(\mu ,{\sigma }^2)$, $\sigma >0$ fixed, every estimator has ${\sup }_{\mu }{\mathbb{E}}_{\mu }|T-\mu |\ge \sigma /(4\sqrt{2n})$. Compare ${\mu }_0=0$ and ${\mu }_1=\sigma /\sqrt{2n}$: the joint Kullback-Leibler divergence is $1/4$, so the total variation–Hellinger–relative entropy inequality bounds total variation distance by $1/2$. Apply the Le Cam lower bound under absolute-error loss. Degenerate zero-noise laws do not satisfy a positive lower bound.

For fixed-design nonparametric regression, compare $m_0=0$ with $m_1(x)=L(h-|x-x_0|{)}_{+}$. This pair has parameter separation $Lh$ at $x_0$, Lipschitz bound $L$, and Kullback-Leibler divergence at most $L^2{h}^2(2nh+1)/(2{\sigma }^2)$. Choosing $h\asymp ({\sigma }^2/(n{L}^2){)}^{1/3}$ controls total variation distance by Pinsker's inequality and yields the pointwise minimax rate for Lipschitz regression through the Le Cam two-point lemma.

Under zero-one loss, a Bayes classifier assigns an observation to a class of greatest posterior probability. For two classes with prior probabilities ${\pi }_0,{\pi }_1$ and densities $f_0,f_1$, it chooses class one exactly when ${\pi }_1{f}_1(x)\ge {\pi }_0{f}_0(x)$.

For two multivariate normal densities, the Bayes log posterior odds are a quadratic polynomial. Equal covariance matrices cancel the quadratic term and give linear discriminant analysis; unequal covariance matrices give quadratic discriminant analysis.

A binary Bayes classifier under zero-one loss is unique up to null sets when the posterior class probabilities tie only on a null set. For two distinct nonsingular Gaussian distributions, the tie set is the zero set of a nonzero quadratic polynomial and has Lebesgue measure zero.

For a model $\{P_{\theta }:\theta \in \Theta \}$, loss $L$, and decision rule $\delta$, the risk is

For a prior $\pi$, the integrated risk is $r(\pi ,\delta )=\int R(\theta ,\delta )\pi (d\theta )$. Its infimum over decision rules is the Bayes risk, attained by a Bayes rule when one exists.

A least favorable prior maximizes the Bayes risk over the allowed priors. If a Bayes rule has constant risk equal to the minimax value, its prior is least favorable.

A minimax rule minimizes the worst-case risk ${\sup }_{\theta \in \Theta }R(\theta ,\delta )$.

An equalizer rule has the same risk at every parameter value. A Bayes equalizer rule is minimax when its constant risk bounds from below the worst-case risk of every competing rule.

For two distinct nonsingular Gaussian class distributions, vary the class-zero prior continuously. The class-zero and class-one error probabilities of the corresponding Bayes classifier cross. At a crossing they are equal, making the classifier an equalizer rule, a minimax decision rule, and the associated prior a least favorable prior.

If a rule has constant risk $r$ and a sequence of priors has Bayes risks tending to $r$, then the rule is minimax. Every competing rule has worst-case risk at least every one of those Bayes risks.

For independent $X_i\sim N(\theta ,1)$ with $\theta \ge 0$, the sample mean has constant squared-error risk $1/n$ and is minimax. Smooth priors supported on $[0,L]$ whose prior Fisher information is $O(L^{-2})$ give, through the Van Trees inequality, a lower bound tending to $1/n$ for every rule's worst-case risk.