Multiple hypothesis testing

Multiple hypothesis testing applies tests to a family of null hypotheses while controlling an aggregate error criterion.

For a nonempty finite index set $I$, this hypothesis asserts that every member $H_i$ is true simultaneously. A valid local test must have its stated size under every parameter satisfying the entire intersection. The Bonferroni correction within $I$ always supplies such a test from valid elementary p-values.

The closure is the collection of all nonempty intersection hypotheses from the original elementary family, including singletons. It is not a topological closure. The closed testing procedure requires local level-$\alpha$ tests for these intersections and rejects an elementary hypothesis only when all intersections containing it reject.

Order $m$ p-values and reject the first $r$ hypotheses, where $r=\max \{j:p_{(j)}\le \alpha /(m-j+1)\}$, with $r=0$ for an empty set. Under independent valid true-null p-values, this step-up procedure controls the familywise error rate. One proof embeds its rejections in the closed testing procedure with Simes tests: every intersection containing a rejected hypothesis passes its local Simes test, so closed-testing control of the familywise error rate applies.

For $m\ge 1$ independent valid null p-values, the displayed inequality follows by applying the Benjamini-Hochberg procedure with all nulls true. Its false discovery rate then equals the probability of at least one rejection. Valid means $\mathrm{Pr}(p_i\le t)\le t$ for $0\le t\le 1$; independence is a sufficient hypothesis, and arbitrary dependence is not covered by this proof.

For the intersection of $m$ null hypotheses, reject when ${\min }_{i}m{p}_{(i)}/i\le \alpha$. The Simes inequality establishes a level-$\alpha$ test under independence of valid true-null p-values. The displayed quantity is its combined p-value.

Given valid p-values $q_J$ for constancy on intervals of length at least two, reject $I$ when ${\max }_{J\supseteq I}n{q}_J/|J|\le \alpha$. Every rejected true interval forces rejection of its maximal constant block. A union bound over the non-singleton maximal blocks gives familywise error rate at most $\alpha$, since their lengths sum to at most $n$. Singleton blocks have no tested interval in this formulation.

For a null hypothesis that is a union $H_0={\bigcup }_{i\in I}{H}_i$, an intersection-union test rejects $H_0$ only when every component $H_i$ is rejected. If each component test has level $\alpha$, then under any point of $H_0$ at least one component null is true, so the combined test also has level at most $\alpha$.

The Bonferroni correction tests each of $m$ hypotheses at level $\alpha /m$. The union bound then controls the familywise error rate by $\alpha$ under arbitrary dependence.

For positive deterministic weights $w_i$, the weighted Bonferroni correction rejects $H_i$ whenThe union bound over the true nulls controls the familywise error rate without an independence assumption.

Order $q_i=p_i/w_i$ increasingly. At rank $k$, reject the corresponding hypothesis whenand otherwise stop. The first ordered true null reduces FWER control to a weighted union bound; the shrinking denominator makes this procedure at least as powerful as the Weighted Bonferroni correction.

The Holm–Bonferroni method orders p-values and compares the $j$th smallest with $\alpha /(m-j+1)$, stopping at the first failed comparison. It controls the familywise error rate under arbitrary dependence.

If the Holm step-down procedure makes a false rejection, it must reject the first true null in the ordered list. All $m_0$ true nulls remain at that step, so its cutoff is at most $\alpha /m_0$. The union bound on their marginally valid p-values gives familywise error rate at most $\alpha$. This proves strong control under arbitrary dependence, and super-uniformity suffices.

Order $m$ p-values as $p_{(1)}\le \cdots \le p_{(m)}$ and take the largest $k$ with $p_{(k)}\le \alpha k/m$. The Benjamini-Hochberg procedure rejects the hypotheses corresponding to $p_{(1)},\dots ,p_{(k)}$ and controls the false discovery rate under independence and several forms of positive dependence.

Let $R$ be the number of rejections by the Benjamini-Hochberg procedure at level $\alpha$, and let $R_i$ be the rejection count after replacing $p_i$ by zero. Then $R_i\ge 1$ andIf $i$ was rejected, lowering its p-value does not change any ordered p-value beyond rank $R$, so $R_i=R$. Conversely, if $p_i\le \alpha R_i/m$, the other $R_i-1$ rejected values together with $p_i$ satisfy the original threshold; monotonicity gives $R=R_i$. For a valid true-null p-value independent of the others, conditioning on $R_i$ bounds the expectation of the right side by $\alpha /m$. Summing over true nulls proves false discovery rate at most $m_0\alpha /m$.

For the Benjamini-Hochberg procedure, replace a true-null p-value $P_i$ by zero and call the new rejection count $R^{(i)}$. The Benjamini-Hochberg leave-one-out identity identifies rejection with $R=r$ with the event $P_i\le \alpha r/m$ and $R^{(i)}=r$. If $P_i$ is uniform and independent of all the other p-values, each possible rejection count contributes $\alpha \mathbb{P}(R^{(i)}=r)/m$ to its expected false-discovery fraction. Summation gives $\alpha /m$ per true null and the displayed exact false discovery rate. Super-uniform independent nulls yield the corresponding upper bound.

The false discovery rate is $\mathbb{E}[V/(R\vee 1)]$, where $V$ is the number of rejected true null hypotheses and $R$ is the total number of rejections.

The familywise error rate is the probability of making at least one false rejection in a family of hypothesis tests.

For nonempty index sets forming a laminar family of sets, reject an intersection hypothesis when the displayed adjusted value is at most $\alpha$. The inclusion-maximal true sets are disjoint and have total size at most $m$. Any false rejection forces some maximal true $J$ to have $p_J\le \alpha |J|/m$. Validity and the union bound control the familywise error rate by $\alpha$, without independence or a complete closure.

Choose a level-$\alpha$ local test for every nonempty intersection $H_I={\bigcap }_{i\in I}{H}_i$. The closed testing procedure rejects an elementary hypothesis $H_i$ exactly when every intersection hypothesis containing it is rejected by its local test.

If any true elementary hypothesis is rejected by a closed testing procedure, then the intersection of all true null hypotheses must also be rejected. Its local test has level $\alpha$, so the familywise error rate is at most $\alpha$ under arbitrary dependence.