= Multiple hypothesis testing
{wiki}
= Multiple testing
{synonym}
<Multiple hypothesis testing> considers a family of <null hypotheses> and a procedure for the collection of rejection decisions. Even if every individual test controls its <type I error>, the collection can produce many false positives. For $m$ independent true nulls, each rejected with <probability> $\alpha$, the <familywise error rate> is
$$
1-(1-\alpha)^m.
$$
For $m=20$ and $\alpha=0.05$, this is about $0.642$. Dependence changes the formula, so <independence> must not be silently assumed.
Different goals require different error criteria. The <familywise error rate> is the <probability> of any false rejection. The <false discovery rate> is the expected fraction of rejected hypotheses that are true, with fraction zero if nothing is rejected. These do not give interchangeable guarantees.
The <Bonferroni correction> tests each of $m$ hypotheses at level $\alpha/m$. If $I_0$ is the set of true nulls, the <union bound> gives
$$
P(\text{any false rejection})\le\sum_{j\in I_0}P(\text{reject }j)\le |I_0|\alpha/m\le\alpha,
$$
without <independence>. <Holm–Bonferroni method> improves this familywise procedure, whereas the <Benjamini-Hochberg procedure> targets <false discovery rate> under its dependence assumptions. Specifying the tested family and intended error criterion is part of the scientific analysis.
Back to article page