Past exam of the mathematics course of the University of Cambridge / 2017 / iii / Paper 207 / 2 / a

A Type I error is rejecting a true null hypothesis. Here it means declaring a statistical association between the biomarker and bowel cancer when, in the population and under the association model being tested, there is no statistical association. It is a false positive association finding, not merely a large estimation error. A test with level $0.05$ limits this probability to $0.05$ under the null hypothesis, subject to valid calibration; the p-value is not the probability that the null hypothesis is true.

Under the intended additional assumption of mutually independent, continuously calibrated null p-values, each test fails to reject with probability $0.95$. The familywise error rate is thereforeThere is a qualification to the printed use of “not correlated”: zero pairwise correlation coefficients do not generally imply mutual independence. A multivariate normal distribution with zero off-diagonal covariances does give independence, but an arbitrary biomarker distribution need not. The formula above requires independent rejection events, not just uncorrelated measurements.

The usual intuition for positively dependent tests is that they reject together, reducing the effective number of opportunities for a false positive. A rigorous sufficient assumption is positive association of random variables for the rejection indicators $I_1,\dots ,I_{10}$. Products of their non-rejection indicators are decreasing functions; association and induction giveThus under this stronger positive-dependence assumption the probability is no larger, and perfect dependence gives $0.05$ instead of $0.4013$. Strict inequality is not forced by every form of positive dependence.

Pairwise positive correlation coefficients alone, as printed, is insufficient. For a counterexample choose a vector of ten exchangeable random variables that are rejection indicators as follows: with probabilities $0.545,0.450,0.005$, respectively, reject none, reject one uniformly selected test, or reject all ten. Thenbut the familywise error rate is $0.455>1-0.95^{10}$. This can be realized with valid null p-values: conditional on the indicators, independently draw $p_i$ uniformly on $(0,0.05)$ if $I_i=1$ and on $(0.05,1)$ otherwise. Every $p_i$ is uniform on $(0,1)$, and $\mathrm{Cov}(p_i,p_j)=0.25\mathrm{Cov}(I_i,I_j)>0$. Taking biomarker statistics $X_i=1-p_i$ gives pairwise positively correlated null statistics with exactly these one-sided tests. Therefore the unconditional larger-or-smaller claim needs a specified dependence model.

The Bonferroni correction compares every p-value with $0.05/100000=5\times 10^{-7}$. Only the first two displayed values lie below that threshold; all undisplayed values are at least the tenth displayed value. HenceFor the Benjamini-Hochberg procedure at false discovery rate level $q=0.1$, the rank-$i$ threshold is $qi/100000=i\times 10^{-6}$. The first nine displayed ranks meet their thresholds, while rank ten does not: $1.1\times 10^{-5}>10^{-5}$. Consequently nine is the intended count if every later rank also fails its threshold. The rule is a step-up rule, however:It does not stop at the first failed comparison. The full rejection count is therefore not determined by ten order statistics. For example, completing the list with $99990$ values equal to $1$ gives $k=9$, whereas completing it with $99990$ values equal to $0.02$ gives $k=100000$, since $p_{(100000)}=0.02\le 0.1$. Both completions respect the printed ten smallest p-values. Thus at least nine are rejected; exactly nine requires an extra assumption about the undisplayed ranks. In the second completion even the tenth marker is rejected, despite failing its own rank threshold. The false discovery rate guarantee also requires the usual validity and independence or appropriate positive-dependence assumptions, unlike the arbitrary-dependence Bonferroni correction.