Statistical decision theory

Statistical decision theory compares decision rules through the expected loss they incur under each parameter value.

If sample transformations preserve the model and both hypotheses, the invariance principle restricts attention to tests whose decisions are unchanged by those transformations. Such tests are functions of a maximal invariant. In estimation, the analogous requirement is equivariance, accompanied by an invariant loss function so that risks transform compatibly. The principle respects symmetries; it does not by itself prove optimality among all unrestricted procedures.

Cost-effectiveness analysis compares interventions through their costs and effects measured in a common outcome unit. Let $\Delta C$ and $\Delta E$ denote mean differences relative to a specified comparator. A positive effect increment with nonpositive cost increment gives dominance; the reverse signs give a dominated intervention. A tradeoff in the positive-positive quadrant requires a willingness-to-pay threshold per outcome unit. Randomized controlled trials can estimate these differences, but their joint sampling distribution is needed to express uncertainty. Outcome units, time horizon and costing perspective must be specified; a survival-day increment is not automatically a quality-adjusted outcome.

At a specified willingness-to-pay threshold $\lambda$, the incremental net monetary benefit is the value of the effect increment minus the cost increment. It avoids dividing by a potentially near-zero or negative effect estimate. For estimates with variances $v_E,v_C$ and covariance $v_{CE}$, its estimated variance is ${\lambda }^2{v}_E+v_C-2\lambda v_{CE}$. A normal approximation therefore gives a confidence interval directly on the net-benefit scale. The economic sign criterion does not require a positive effect increment.

A cost-effectiveness acceptability curve displays the probability of positive incremental net monetary benefit as a function of the willingness-to-pay threshold. Under a posterior distribution this is a posterior probability. A frequentist paired bootstrap version displays the fraction of replicates for which the net benefit is positive, a resampling measure of uncertainty rather than literally a probability that a fixed true parameter is positive. Using the sign of net benefit handles all quadrants of the cost-effectiveness plane; counting only replicates whose ratio is below the threshold does not.

The cost-effectiveness plane plots an effect increment horizontally and a cost increment vertically. The southeast quadrant represents greater effectiveness at lower cost, while the northwest quadrant represents lower effectiveness at greater cost. A threshold $\lambda$ corresponds to the line $\Delta C=\lambda \Delta E$; points below the line have positive incremental net monetary benefit, in every quadrant. A joint confidence region or paired bootstrap cloud reveals uncertainty and preserves the dependence between cost and effect estimates.

The incremental cost-effectiveness ratio divides an intervention's additional mean cost by its additional mean effect relative to a comparator. Its units are cost per outcome unit. When both increments are positive, comparison with a threshold $\lambda$ is equivalent to checking whether the incremental net monetary benefit $\lambda \Delta E-\Delta C$ is positive. When $\Delta E<0$ the inequality reverses on division, and when $\Delta E=0$ the ratio is undefined. A ratio alone cannot identify the quadrant of the cost-effectiveness plane.

A prior on a parameter, a conditional observation distribution, an action space and a loss specify a Bayesian decision problem. The normal form minimizes prior expected loss over decision rules; the extensive form minimizes posterior expected loss after each observation. Conditional expectation gives the same Bayes risk for these formulations, provided measurable optimal or approximate choices can be made.

A measurable choice of an action from observed data, with a randomized rule allowing a conditional distribution of actions. Its performance under a loss function is the risk function $R(\theta ,\delta )={\mathbb{E}}_{\theta }L(\theta ,\delta (X))$. Admissibility compares risks pointwise, Bayes optimality integrates risk under a prior, and minimax optimality minimizes the worst-case risk.

The loss $|a-\theta |$ for estimating a real parameter $\theta$ by $a$. Its risk of a decision rule is the corresponding expected value. Unlike squared-error loss, its pointwise size is a metric, so the triangle inequality gives a direct overlap version of the Le Cam two-point lemma.

This loss assigns different positive costs to underestimation and overestimation of a scalar parameter. Here $\gamma$ penalizes estimates below the parameter and $\delta$ estimates above it. Equal costs give a multiple of absolute-error loss.

For a posterior distribution with finite first moment, the posterior risk derivative at continuity points is $(\gamma +\delta )F(a)-\gamma$. Convexity therefore makes a posterior $\gamma /(\gamma +\delta )$-quantile a Bayes estimator. With atoms, the condition is $F(a^{-})\le \gamma /(\gamma +\delta )\le F(a)$. Increasing the cost of underestimation moves the optimal estimate upward.

Quadratic loss penalizes an estimate $a$ of a vector parameter $\theta$ by $\Vert a-\theta {\Vert }^2$. Its posterior expected value is minimized by the posterior mean.

The risk of an estimator under squared Euclidean error is its expected squared distance from the true parameter. Pointwise risk domination can be strict even when the supremum risk agrees, because the supremum can be approached at parameters tending to infinity.

The mean-vector prediction risk measures squared Euclidean norm error in estimating the deterministic response mean. For independent future noise with covariance matrix ${\sigma }^2{I}_n$, prediction of a new noisy response adds $n{\sigma }^2$ to this risk. A Gaussian orthogonal projection matrix $P$ of rank $k$ has risk $\Vert (I-P)\mu {\Vert }^2+k{\sigma }^2$.

For $Y\sim N_n(\mu ,{\sigma }^{2}I)$ and a fixed orthogonal projection matrix $P$ of rank $k$, the displayed expression is an unbiased estimator of the mean-vector prediction risk whenever ${\hat{\sigma }}^2$ is an unbiased estimator of ${\sigma }^2$. Independence of its two terms is unnecessary. Comparing fixed models yields the Mallows Cp penalty, but minimizing unbiased estimates does not preserve unbiasedness after selection.

For a normal distribution $N_n(\mu ,{\sigma }^{2}I)$ with unrestricted $\mu \in {\mathbb{R}}^n$ and unknown ${\sigma }^2$, an integrable data-only unbiased estimator of the mean-vector prediction risk of a fixed rank-$k$ orthogonal projection matrix exists exactly when $2k=n$. To prove necessity, randomize the mean by independent Gaussian variance $v$: conditioning adds $(n-k)v$ to the squared bias term, whereas the marginal variance identity would add $kv$. When $2k=n$, the residual sum of squares itself is unbiased.

Le Cam's two-point lemma lower-bounds minimax risk by reducing estimation to testing two separated parameter values. Under squared loss, one form is

If two parameters are separated by at least $2r$ in a metric, any estimator gives a test by deciding whether its distance from the first parameter is below $r$. For $0<\eta <1$, the event $|Z-1|\le \eta$ and Markov inequality give the displayed minimax risk bound, where $Z$ is the product likelihood ratio. The unrestricted claim for every positive $\eta$ is false when both factors on the right are negative.

For any estimator $T$ in a metric parameter space, $d(T,f{)}^2+d(T,g{)}^2\ge d(f,g{)}^2/2$ by the triangle inequality. Average the two risk functions and replace their probability density functions by their minimum. This proves $\max (R_f,R_g)\ge d(f,g{)}^2\int \min (p_f,p_g)/4$. The overlap equals one minus total variation distance. The same proof applies to squared error for a real-valued statistical functional of the parameter, without requiring that functional to be injective.

For estimating the expected value over all continuous probability density functions on $[0,1]$, compare $f=1$ and $g_n(u)=1+(u-1/2)/\sqrt{n}$. Their means differ by $1/(12\sqrt{n})$, and ${\chi }^2(P_{g_n}\Vert P_f)=1/(12n)$. The chi-squared divergence of product measures bounds joint total variation distance by $1/2$. The metric squared-loss two-point bound then gives the displayed uniform minimax risk lower bound.

The Cauchy-Schwarz inequality gives $\Vert P-Q{\Vert }_{\mathrm{TV}}=\frac{1}{2}{\mathbb{E}}_Q|dP/dQ-1|\le \frac{1}{2}\sqrt{{\chi }^2(P\Vert Q)}$. Every statistical hypothesis testing rule has sum of its Type I error and Type II error at least $1-\Vert P-Q{\Vert }_{\mathrm{TV}}$, proving the displayed bound. A mixture model of alternatives yields a lower bound on the worst alternative error because a maximum dominates an average.

For two observation laws $P_0,P_1$ at real parameters separated by $\Delta$, every estimator $T$ satisfies ${\max }_j{\mathbb{E}}_j|T-{\theta }_j|\ge \Delta (1-\mathrm{TV}(P_0,P_1))/2$. With common densities, the sum of the two risks is at least $\int \min (p_0,p_1)(|T-{\theta }_0|+|T-{\theta }_1|)$, at least $\Delta \int \min (p_0,p_1)=\Delta (1-\mathrm{TV})$ by the triangle inequality. Divide by two. The same proof works for any metric loss.

For $n$ independent observations with a normal distribution $N(\mu ,{\sigma }^2)$, $\sigma >0$ fixed, every estimator has ${\sup }_{\mu }{\mathbb{E}}_{\mu }|T-\mu |\ge \sigma /(4\sqrt{2n})$. Compare ${\mu }_0=0$ and ${\mu }_1=\sigma /\sqrt{2n}$: the joint Kullback-Leibler divergence is $1/4$, so the total variation–Hellinger–relative entropy inequality bounds total variation distance by $1/2$. Apply the Le Cam lower bound under absolute-error loss. Degenerate zero-noise laws do not satisfy a positive lower bound.

For fixed-design nonparametric regression, compare $m_0=0$ with $m_1(x)=L(h-|x-x_0|{)}_{+}$. This pair has parameter separation $Lh$ at $x_0$, Lipschitz bound $L$, and Kullback-Leibler divergence at most $L^2{h}^2(2nh+1)/(2{\sigma }^2)$. Choosing $h\asymp ({\sigma }^2/(n{L}^2){)}^{1/3}$ controls total variation distance by Pinsker's inequality and yields the pointwise minimax rate for Lipschitz regression through the Le Cam two-point lemma.

Under zero-one loss, a Bayes classifier assigns an observation to a class of greatest posterior probability. For two classes with prior probabilities ${\pi }_0,{\pi }_1$ and densities $f_0,f_1$, it chooses class one exactly when ${\pi }_1{f}_1(x)\ge {\pi }_0{f}_0(x)$.

With zero cost for correct classifications, assign class one when the displayed inequality holds and class two otherwise. It compares conditional expected losses; ties may be resolved arbitrarily. For positive costs and densities, the log likelihood ratio threshold is $\log (c(1\mid 2){\pi }_2/(c(2\mid 1){\pi }_1))$.

For integrable real $g$, including points where $g<0$ decreases the integral and including points where $g>0$ increases it. Thus the negative set minimizes the integral, up to null sets and arbitrary choices where $g=0$. This elementary signed-integral fact derives the Bayes classifier and cost-sensitive Bayes classifier.

For two multivariate normal densities, the Bayes log posterior odds are a quadratic polynomial. Equal covariance matrices cancel the quadratic term and give linear discriminant analysis; unequal covariance matrices give quadratic discriminant analysis.

For two multivariate normal classes with a common positive-definite covariance matrix, $L={\Sigma }^{-1}({\mu }_1-{\mu }_2)$ and the displayed score is the log posterior odds of class one against class two. With equal error costs, positive scores favor class one. Changing the prior odds shifts the separating hyperplane parallel to itself, expanding the region assigned to the more frequent class. On the boundary either decision has the same conditional risk.

For two multivariate normal classes with equal priors and common positive-definite covariance, the optimal score is $({\mu }_1-{\mu }_2{)}^T{\Sigma }^{-1}(x-({\mu }_1+{\mu }_2)/2)$. Its distribution within either class is normal with variance $D^2$ and mean respectively $D^2/2$ or $-D^2/2$. Both class error probabilities are therefore the displayed normal tail.

A binary Bayes classifier under zero-one loss is unique up to null sets when the posterior class probabilities tie only on a null set. For two distinct nonsingular Gaussian distributions, the tie set is the zero set of a nonzero quadratic polynomial and has Lebesgue measure zero.

For a model $\{P_{\theta }:\theta \in \Theta \}$, loss $L$, and decision rule $\delta$, the risk is

For a prior $\pi$, the integrated risk is $r(\pi ,\delta )=\int R(\theta ,\delta )\pi (d\theta )$. Its infimum over decision rules is the Bayes risk, attained by a Bayes rule when one exists.

A Bayes act minimizes $\int L(\theta ,a)\Pi (d\theta )$ for a specified parameter distribution $\Pi$. Applying the same minimization to the posterior after observing data gives a Bayes decision rule. The optimal data-based Bayes risk is the expectation of the optimal posterior no-data loss.

For a probability density $\pi$ and threshold $c>0$, this region includes precisely the points at or above the threshold. It is a Bayes act for loss $c|S|-1_S(\theta )$, since expected loss is ${\int }_S(c-\pi (\theta ))d\theta$. The minimal value is $-\int (\pi -c{)}_{+}d\theta$; adding or removing threshold-equality points does not change it.

A least favorable prior maximizes the Bayes risk over the allowed priors. If a Bayes rule has constant risk equal to the minimax value, its prior is least favorable.

A minimax rule minimizes the worst-case risk ${\sup }_{\theta \in \Theta }R(\theta ,\delta )$.

An equalizer rule has the same risk at every parameter value. A Bayes equalizer rule is minimax when its constant risk bounds from below the worst-case risk of every competing rule.

For two distinct nonsingular Gaussian class distributions, vary the class-zero prior continuously. The class-zero and class-one error probabilities of the corresponding Bayes classifier cross. At a crossing they are equal, making the classifier an equalizer rule, a minimax decision rule, and the associated prior a least favorable prior.

If a rule has constant risk $r$ and a sequence of priors has Bayes risks tending to $r$, then the rule is minimax. Every competing rule has worst-case risk at least every one of those Bayes risks.

For independent $X_i\sim N(\theta ,1)$ with $\theta \ge 0$, the sample mean has constant squared-error risk $1/n$ and is minimax. Smooth priors supported on $[0,L]$ whose prior Fisher information is $O(L^{-2})$ give, through the Van Trees inequality, a lower bound tending to $1/n$ for every rule's worst-case risk.