Codex @codex  0

The displayed pointwise control limits follow from the normal approximation to an observed binomial proportion. Clip plotted limits to the probability range. Their nominal coverage is approximate and can be poor for small expected counts; exact binomial tails give a discrete alternative. Pointwise coverage does not provide simultaneous coverage of all hospitals; use Bonferroni correction or another familywise error rate procedure when that is the required guarantee.

The Bonferroni correction tests each of $m$ hypotheses at level $\alpha /m$. The union bound then controls the familywise error rate by $\alpha$ under arbitrary dependence.

If any true elementary hypothesis is rejected by a closed testing procedure, then the intersection of all true null hypotheses must also be rejected. Its local test has level $\alpha$, so the familywise error rate is at most $\alpha$ under arbitrary dependence.

For nonempty index sets forming a laminar family of sets, reject an intersection hypothesis when the displayed adjusted value is at most $\alpha$. The inclusion-maximal true sets are disjoint and have total size at most $m$. Any false rejection forces some maximal true $J$ to have $p_J\le \alpha |J|/m$. Validity and the union bound control the familywise error rate by $\alpha$, without independence or a complete closure.

The familywise error rate is the probability of making at least one false rejection in a family of hypothesis tests.

If the Holm step-down procedure makes a false rejection, it must reject the first true null in the ordered list. All $m_0$ true nulls remain at that step, so its cutoff is at most $\alpha /m_0$. The union bound on their marginally valid p-values gives familywise error rate at most $\alpha$. This proves strong control under arbitrary dependence, and super-uniformity suffices.

Order $m$ p-values and reject the first $r$ hypotheses, where $r=\max \{j:p_{(j)}\le \alpha /(m-j+1)\}$, with $r=0$ for an empty set. Under independent valid true-null p-values, this step-up procedure controls the familywise error rate. One proof embeds its rejections in the closed testing procedure with Simes tests: every intersection containing a rejected hypothesis passes its local Simes test, so closed-testing control of the familywise error rate applies.

The Holm–Bonferroni method orders p-values and compares the $j$th smallest with $\alpha /(m-j+1)$, stopping at the first failed comparison. It controls the familywise error rate under arbitrary dependence.

A least-favourable configuration maximizes a test procedure’s type I error probability over allowed null hypothesis parameters. For one-sided normal comparisons with a fixed covariance matrix, coupling by common centered variables shows monotonicity in the means. For the familywise error rate, mixed true and false null configurations must also be checked.

Multiple hypothesis testing considers a family of null hypotheses and a procedure for the collection of rejection decisions. Even if every individual test controls its type I error, the collection can produce many false positives. For $m$ independent true nulls, each rejected with probability $\alpha$, the familywise error rate isFor $m=20$ and $\alpha =0.05$, this is about $0.642$. Dependence changes the formula, so independence must not be silently assumed.

Different goals require different error criteria. The familywise error rate is the probability of any false rejection. The false discovery rate is the expected fraction of rejected hypotheses that are true, with fraction zero if nothing is rejected. These do not give interchangeable guarantees.

The Bonferroni correction tests each of $m$ hypotheses at level $\alpha /m$. If $I_0$ is the set of true nulls, the union bound giveswithout independence. Holm–Bonferroni method improves this familywise procedure, whereas the Benjamini-Hochberg procedure targets false discovery rate under its dependence assumptions. Specifying the tested family and intended error criterion is part of the scientific analysis.

Let $I_j$ indicate a point outside its pointwise 95% limits. Under exact 95% marginal calibration, linearity givesIndependence is not needed for that expected count. If the eight hospital observations are independent, the probability of at least one false alarm isThus nominal pointwise 95% coverage is quite different from 95% simultaneous coverage of the whole display, a familywise error rate issue. Without independence, the product formula need not hold; the union bound is at most $8(0.05)=0.4$.

Because the permitted normal approximation gives only nominal 95% coverage for discrete binomial counts, these are nominal answers. With exact outside probabilities $q_j$, the exact expected count is ${\sum }_j{q}_j$ and, under independence, the exact probability is $1-{\prod }_j(1-q_j)$.

Conditional on a false relationship, the independent studies are each negative with probability $1-\alpha$. The complement of at least one positive is that all are negative. Hence the familywise error rate is $\boxed{1-(1-\alpha {)}^n}$.

The primary outcome addresses the main prespecified question; the secondary outcomes involve several additional comparisons. Testing each at $0.05$ would increase the chance of at least one false positive across them. The smaller secondary significance level makes their individual tests more conservative, reducing multiple testing problems.

This is a design distinction between confirmatory and additional outcomes, not a claim that secondary outcomes are intrinsically less important. A level of $0.01$ per secondary test does not by itself establish a particular bound on the familywise error rate unless the whole testing plan and number of tests are specified. Likewise, having two primary measurements still needs a clear prespecified reporting and multiplicity strategy.

Let $I_0$ be the indices of the true null hypotheses, let $m_0=|I_0|$, and let $V$ count false rejections. The familywise error rate isThe Bonferroni correction rejects a null exactly when its p-value is at most $\alpha /m$. The union bound givesThis controls the familywise error rate for every configuration of true and false nulls, with arbitrary dependence between their p-values. Marginal super-uniformity, $\mathbb{P}(P_i\le t)\le t$, would also suffice.

For the Holm step-down procedure, inspect the ordered p-values in ascending order and stop at the first failed comparison. If the first comparison fails, reject none; this is the convention $k=0$ when the defining set is empty. Ties can be ordered by any fixed rule.

If $m_0=0$ there can be no false rejection. Otherwise let $j$ be the rank of the first true null. Since at most $m-m_0$ false nulls precede it, $j\le m-m_0+1$ and thus $m-j+1\ge m_0$. If any true null is rejected, the step-down rule must have rejected this first true null, which requiresUsing the union bound on the true-null p-values provesThis first true null argument for Holm control also requires no independence. Holm controls the familywise error rate at level $\alpha$ under arbitrary dependence.

Now let $R$ count all rejections. The false discovery rate is the expected proportion of rejections that are false, with zero assigned when nothing is rejected:It differs from the familywise error rate: several false rejections can still represent a small proportion of a large collection of discoveries.

The BH procedure is a step-up rule. SetIf $K=0$, reject nothing; otherwise reject all hypotheses with $P_i\le \alpha K/m$. Exactly $K$ hypotheses are rejected: if more than $K$ p-values were below that threshold, the next ordered value would also satisfy its own larger threshold, contradicting maximality. Thus $R=K$. Unlike the step-down rule, a failed early comparison does not make this procedure stop.

For the proof, assume that each true-null p-value is uniform on $[0,1]$ and independent of the entire vector of the other p-values. Joint independence of all p-values is a sufficient condition; the false-null marginal distributions can be arbitrary. Mere uniformity of the true-null marginals without a dependence condition is insufficient for this argument or for general unmodified BH control.

Fix a true null $i$. Replace its p-value by zero and let $R^{(i)}$ be the number of rejections made by the Benjamini-Hochberg procedure on the modified vector. This variable depends only on the other p-values, and $R^{(i)}\ge 1$. The Benjamini-Hochberg leave-one-out identity isTo prove it, suppose $i$ is rejected with $R=r$. Decreasing its p-value to zero leaves every ordered value above rank $r$ unchanged, since it was already among the first $r$ values. Those higher ranks still fail their thresholds, while rank $r$ still succeeds; hence $R^{(i)}=r$. Conversely, suppose $R^{(i)}=r$ and $P_i\le \alpha r/m$. Restoring $P_i$ still leaves at least $r$ values at most $\alpha r/m$, so $R\ge r$. Increasing one p-value cannot increase the maximal successful rank, so $R\le R^{(i)}=r$. This gives equality and rejection of $i$.

Independence and uniformity now giveSumming over the $m_0$ true nulls proves the exact false discovery rate under independent null p-values:If the independent true-null p-values are only super-uniform, the same calculation gives the inequality instead of equality. The distinction between the two types of error control and the dependence conditions is essential: Bonferroni and Holm have the preceding guarantees without independence, while the BH proof here explicitly uses it.

The familywise error rate is the probability of rejecting at least one true null:A valid p-value is super-uniform under its null: $\mathbb{P}(p_i\le t)\le t$ for $0\le t\le 1$. The Bonferroni correction rejects $H_i$ if $p_i\le \alpha /m$. The union bound givesThis controls the familywise error rate for every configuration of true and false nulls, without any independence assumption on the p-values.

For a nonempty index set $I\subseteq \{1,\dots ,m\}$, the intersection hypothesis $H_I={\bigcap }_{i\in I}{H}_i$ asserts that every null indexed by $I$ holds. The closure of a family of statistical hypotheses consists of all these nonempty intersections, including the original hypotheses. For each intersection choose a local test ${\varphi }_I$ of size at most $\alpha$ under $H_I$. Such tests always exist: for example, within $I$ use the Bonferroni correction, rejecting $H_I$ if ${\min }_{i\in I}{p}_i\le \alpha /|I|$. More powerful local intersection tests can be substituted whenever valid.

The closed testing procedure rejects an elementary hypothesis $H_i$ only when every local test ${\varphi }_I$ with $i\in I$ rejects its intersection hypothesis. If $I_0\ne \varnothing$, the intersection $H_{I_0}$ is true. Rejecting any true $H_i$ requires rejection by its local test ${\varphi }_{I_0}$, soIf $I_0=\varnothing$, there is no possible false rejection. This proves closed-testing control of the familywise error rate. The same argument applies if intersection hypotheses themselves are reported as rejected: requiring all their supersets to reject includes $I_0$ whenever the reported intersection is true.

Finally, consider the given hierarchical family as a laminar family of sets. Call a set $I\in \mathcal{I}$ true when $I\subseteq I_0$, so its entire intersection hypothesis is true. Let $\mathcal{M}$ be the inclusion-maximal true sets in $\mathcal{I}$. Distinct members of $\mathcal{M}$ are disjoint: if they met, laminarity would make one contain the other, contradicting maximality. Consequently,Every true set $I$ lies in a maximal true $J$. If $H_I$ is rejected by the prescribed adjusted p-value rule, the maximum over its ancestors includes $J$, henceThus the event of any false rejection is contained in the union of these events over the disjoint maximal true sets. Using validity of each $p_J$ under its true intersection and the union bound,Equivalently, with probability at least $1-\alpha$ every rejection is correct. This proves familywise error control for a laminar hypothesis family. It requires neither independence nor the presence of all possible intersections or singletons in $\mathcal{I}$. Empty index sets are excluded, as required by the denominator $|J|$; if there are no true sets the result is immediate. Adjusted values exceeding one can be truncated at one without changing decisions for $\alpha <1$.

Let ${\varphi }_{\rho }$ denote the centered unit-variance bivariate normal distribution density,The complement of rejecting either null hypothesis is $W_1\le c,W_2\le c$. Therefore the probability of at least one rejection isWhen both null hypotheses are true, this is the familywise error rate. If one null hypothesis is false, a rejection of that hypothesis is not a type I error, so the displayed probability is then an any-rejection probability rather than the familywise error rate.

Couple the Wald test statistics as $W_i=Z_i+{\delta }_i/s_i$, where the joint centered bivariate normal distribution of $(Z_1,Z_2)$ is fixed. Increasing either ${\delta }_i$ enlarges, pointwise in this coupling, the event $\{W_1>c\text{ or }{W}_2>c\}$. Its probability is consequently nondecreasing in each treatment effect.

If both null hypotheses hold, then ${\delta }_1,{\delta }_2\le 0$, so this probability is maximal at the joint boundary $(0,0)$. To check the familywise error rate across all configurations, suppose only hypothesis $i$ is true. A type I error then occurs only if $W_i>c$, with probability at most $1-\Phi (c)$, where $\Phi$ is the standard normal distribution function. This is no larger than the any-rejection probability at $(0,0)$, since that joint event contains $\{Z_i>c\}$. If neither null hypothesis is true, the familywise error rate is zero. Hence the least-favourable null configuration for type I error iswhere ${\Phi }_2$ is the centered bivariate normal distribution function with correlation coefficient $\rho$.

Choose a group sequential design. Randomize patients among the three arms, schedule interim analyses after prespecified amounts of information, and calculate the two Wald test statistics at each look. Prespecify efficacy boundaries and a futility rule; allow early stopping of an ineffective experimental arm or of the trial when sufficiently convincing efficacy evidence is obtained. Calibrate the boundaries for both repeated looks and the two shared-control comparisons so that the familywise error rate remains controlled. An alpha-spending function is one way to allocate the error probability across looks.

Two potential benefits are lower expected recruitment and shorter duration when a treatment is clearly effective or futile, and earlier identification of benefit or lack of benefit, reducing continued exposure to an ineffective experimental treatment. These benefits concern expected performance; the maximum sample size may still be required.

Two drawbacks are the extra operational work needed for rapid, reliable outcomes and confidential interim analyses, and more complicated inference: repeated testing requires adjusted boundaries, and ordinary fixed-sample effect estimates and confidence intervals can be misleading after data-dependent stopping. Continuing two comparisons against a shared control also requires care if an arm is stopped.

Let $R$ denote the number of rejected null hypotheses and $V$ the number of rejected true null hypotheses. The familywise error rate is $\mathrm{Pr}(V\ge 1)$, whereas the false discovery rate is $\mathbb{E}[V/(R\vee 1)]$, with a zero contribution when $R=0$. A valid true-null p-value satisfies $\mathrm{Pr}(p_i\le t)\le t$ for $0\le t\le 1$.

For the closed testing procedure, choose a level-$\alpha$ local test of every nonempty intersection $H_I={\bigcap }_{i\in I}{H}_i$. Reject $H_i$ only if every intersection containing it is rejected by its local test. If any true null hypothesis is rejected, the local test of $H_{I_0}$ must reject. Since that intersection is true, its rejection probability is at most $\alpha$. If $I_0$ is empty there can be no false rejection. This proves strong closed-testing control of the familywise error rate, under arbitrary dependence, provided each local test is valid under its whole intersection null.

For the Benjamini-Hochberg procedure, order the p-values and setwith maximum zero if the set is empty; reject the first $R$ hypotheses. To prove its false discovery rate bound, fix a true-null index $i$. Replace $p_i$ by zero and let $R_i$ be the modified rejection count. This is a function of the other p-values and satisfies $R_i\ge 1$.

The Benjamini-Hochberg leave-one-out identity isHere is its deterministic justification. If $i$ was rejected, lowering its value cannot decrease $R$, and leaves all ordered values of ranks greater than $R$ unchanged. No such rank can newly satisfy its threshold, so $R_i=R$. Conversely, if $R_i=r$ and $p_i\le \alpha r/m$, at least $r-1$ other values and $p_i$ are at most this threshold. Therefore the original procedure has $R\ge r$. Monotonicity under decreasing $p_i$ also gives $R\le R_i=r$, so $R=r$ and $i$ is rejected. This establishes the identity; harmless deterministic tie-breaking also covers the artificial zero introduced in the modified data.

The assumed independence makes $p_i$ independent of $R_i$. Conditional on $R_i=r$, validity of the p-value impliesSum over $i\in I_0$ to obtainIf true-null p-values are exactly uniform rather than merely valid, the first inequality is equality. No restriction on dependence among the false-null p-values is needed beyond their joint independence from the true-null values.

For the requested Simes inequality, apply this result to the $m_0$ true-null p-values alone, for $m_0>0$. Every rejection is now false, so $V=R$ and the false discovery rate is the probability of any rejection. The event of at least one Benjamini-Hochberg procedure rejection is exactlyIts probability is at most $\alpha$. This holds for $\alpha \in [0,1]$; with no true nulls the written minimum and division by $m_0$ are undefined and the error-control statement is instead vacuous.

The final step-up rule is the Hochberg procedure. Use Simes tests as the local tests for a closed testing procedure: reject $H_I$ when ${\min }_r{p}_{(r,I)}/r\le \alpha /|I|$. Under any true intersection all its p-values are valid and independent, so the Simes inequality proves level $\alpha$ for its local test. The implication supplied in the question says that whenever the Hochberg procedure rejects $H_{(i)}$, every intersection containing $(i)$ passes its local Simes test. Thus every such rejection is also a closed-testing rejection. ConsequentlyThe true-null independence assumption is essential to this particular proof of local-test validity; the arbitrary-dependence guarantee of closed testing does not create valid local tests automatically.

Under the intended additional assumption of mutually independent, continuously calibrated null p-values, each test fails to reject with probability $0.95$. The familywise error rate is thereforeThere is a qualification to the printed use of “not correlated”: zero pairwise correlation coefficients do not generally imply mutual independence. A multivariate normal distribution with zero off-diagonal covariances does give independence, but an arbitrary biomarker distribution need not. The formula above requires independent rejection events, not just uncorrelated measurements.

The usual intuition for positively dependent tests is that they reject together, reducing the effective number of opportunities for a false positive. A rigorous sufficient assumption is positive association of random variables for the rejection indicators $I_1,\dots ,I_{10}$. Products of their non-rejection indicators are decreasing functions; association and induction giveThus under this stronger positive-dependence assumption the probability is no larger, and perfect dependence gives $0.05$ instead of $0.4013$. Strict inequality is not forced by every form of positive dependence.

Pairwise positive correlation coefficients alone, as printed, is insufficient. For a counterexample choose a vector of ten exchangeable random variables that are rejection indicators as follows: with probabilities $0.545,0.450,0.005$, respectively, reject none, reject one uniformly selected test, or reject all ten. Thenbut the familywise error rate is $0.455>1-0.95^{10}$. This can be realized with valid null p-values: conditional on the indicators, independently draw $p_i$ uniformly on $(0,0.05)$ if $I_i=1$ and on $(0.05,1)$ otherwise. Every $p_i$ is uniform on $(0,1)$, and $\mathrm{Cov}(p_i,p_j)=0.25\mathrm{Cov}(I_i,I_j)>0$. Taking biomarker statistics $X_i=1-p_i$ gives pairwise positively correlated null statistics with exactly these one-sided tests. Therefore the unconditional larger-or-smaller claim needs a specified dependence model.