= Prespecified versus data-selected confidence allocations
\b[Prespecified versus data-selected confidence allocations] distinguishes a fixed procedure's coverage from coverage after choosing among procedures using the same observations. For detection-calibrated prevalence, a fixed pair $(\alpha,\beta)$ with $\alpha+\beta=\gamma$ gives a bound with coverage at least $1-\gamma$ when the two marginal bounds are valid. Taking the smallest result from many such allocations requires another argument: the selected result fails if any candidate producing that small result fails.
An original counterexample makes the issue exact. For an unknown location parameter $\theta$, suppose the observed pair is $X=\theta+U$, $Y=\theta-U$, where $U$ has a <uniform distribution> on $[0,1]$. Two upper <confidence bounds> are
$$
B_1=X-0.05,\qquad B_2=Y+0.95.
$$
Each separately covers $\theta$ with probability 0.95. The first fails when $U<0.05$, and the second when $U>0.95$. Choosing the smaller observed bound gives
$$
\Pr\{\theta\le\min(B_1,B_2)\}=\Pr\{0.05\le U\le0.95\}=0.90.
$$
The selection uses only the reported bounds, yet loses the individual 95% coverage. These artificial observations illustrate a selection effect; they are not a model of actual anatomical calibration data.
For $M$ prespecified candidate allocations, let each candidate's total failure budget be at most $\gamma/M$. The <union bound> then controls the failure of any candidate by at most $\gamma$, so even the selected minimum retains coverage at least $1-\gamma$. This is a direct <Bonferroni correction>; NIST presents the corresponding simultaneous-coverage principle.<NIST and SEMATECH e-Handbook of Statistical Methods>[§7.4.7.3, Bonferroni's method, simultaneous statements and division of the error probability]{ref} For two candidates and a 5% overall budget, each candidate receives at most 2.5%, including both its prevalence and detection-calibration contributions.
A continuum of possible allocations is not covered by assigning each the original finite-candidate budget. <Uniform coverage over confidence-allocation choices> requires a jointly valid construction or a proof that includes the selection rule. A split chosen from independent pilot data can instead be treated as fixed conditional on that pilot, provided the required coverage is valid for every resulting split. The source of the allocation and the data used for the reported bound are therefore part of the statistical procedure.
Back to article page
