Past exam of the mathematics course of the University of Cambridge / 2013 / iii / Paper 31 / 4

Let $I_0$ be the indices of the true null hypotheses, let $m_0=|I_0|$, and let $V$ count false rejections. The familywise error rate isThe Bonferroni correction rejects a null exactly when its p-value is at most $\alpha /m$. The union bound givesThis controls the familywise error rate for every configuration of true and false nulls, with arbitrary dependence between their p-values. Marginal super-uniformity, $\mathbb{P}(P_i\le t)\le t$, would also suffice.

For the Holm step-down procedure, inspect the ordered p-values in ascending order and stop at the first failed comparison. If the first comparison fails, reject none; this is the convention $k=0$ when the defining set is empty. Ties can be ordered by any fixed rule.

If $m_0=0$ there can be no false rejection. Otherwise let $j$ be the rank of the first true null. Since at most $m-m_0$ false nulls precede it, $j\le m-m_0+1$ and thus $m-j+1\ge m_0$. If any true null is rejected, the step-down rule must have rejected this first true null, which requiresUsing the union bound on the true-null p-values provesThis first true null argument for Holm control also requires no independence. Holm controls the familywise error rate at level $\alpha$ under arbitrary dependence.

Now let $R$ count all rejections. The false discovery rate is the expected proportion of rejections that are false, with zero assigned when nothing is rejected:It differs from the familywise error rate: several false rejections can still represent a small proportion of a large collection of discoveries.

The BH procedure is a step-up rule. SetIf $K=0$, reject nothing; otherwise reject all hypotheses with $P_i\le \alpha K/m$. Exactly $K$ hypotheses are rejected: if more than $K$ p-values were below that threshold, the next ordered value would also satisfy its own larger threshold, contradicting maximality. Thus $R=K$. Unlike the step-down rule, a failed early comparison does not make this procedure stop.

For the proof, assume that each true-null p-value is uniform on $[0,1]$ and independent of the entire vector of the other p-values. Joint independence of all p-values is a sufficient condition; the false-null marginal distributions can be arbitrary. Mere uniformity of the true-null marginals without a dependence condition is insufficient for this argument or for general unmodified BH control.

Fix a true null $i$. Replace its p-value by zero and let $R^{(i)}$ be the number of rejections made by the Benjamini-Hochberg procedure on the modified vector. This variable depends only on the other p-values, and $R^{(i)}\ge 1$. The Benjamini-Hochberg leave-one-out identity isTo prove it, suppose $i$ is rejected with $R=r$. Decreasing its p-value to zero leaves every ordered value above rank $r$ unchanged, since it was already among the first $r$ values. Those higher ranks still fail their thresholds, while rank $r$ still succeeds; hence $R^{(i)}=r$. Conversely, suppose $R^{(i)}=r$ and $P_i\le \alpha r/m$. Restoring $P_i$ still leaves at least $r$ values at most $\alpha r/m$, so $R\ge r$. Increasing one p-value cannot increase the maximal successful rank, so $R\le R^{(i)}=r$. This gives equality and rejection of $i$.

Independence and uniformity now giveSumming over the $m_0$ true nulls proves the exact false discovery rate under independent null p-values:If the independent true-null p-values are only super-uniform, the same calculation gives the inequality instead of equality. The distinction between the two types of error control and the dependence conditions is essential: Bonferroni and Holm have the preceding guarantees without independence, while the BH proof here explicitly uses it.