Statistical hypothesis test

Statistical hypothesis testing compares data against a null hypothesis using a controlled rejection probability.

An equivalence test asks whether an effect lies strictly inside a prespecified negligible-effect range, such as $(-\delta ,\delta )$ for $\delta >0$. Its null hypothesis is that the effect lies outside that range, including its boundary; its alternative hypothesis is $-\delta <\theta <\delta$. This differs from a test with a zero-effect null hypothesis: failing to reject zero does not establish that the effect is sufficiently small.

One construction performs two level-$\alpha$ one-sided tests, rejecting $\theta \le -\delta$ and rejecting $\theta \ge \delta$, and declares equivalence only if both reject. If the equivalence null hypothesis is true, at least one component null is true. The probability of rejecting both cannot exceed the probability of rejecting that true component, which is at most $\alpha$. Thus the combined procedure controls Type I error at level $\alpha$. In a continuous symmetric t-based location model with a common standard error, this is equivalent to putting the central $100(1-2\alpha )\%$ confidence interval wholly inside $(-\delta ,\delta )$. At $\alpha =0.05$ that is a 90% interval, not the 95% interval used for the usual two-sided zero-effect test. The margin must be chosen for substantive reasons rather than selected afterward to accommodate an observed interval.

A randomized statistical test specifies a measurable rejection probability $\varphi (x)$ for each possible data value. It can be implemented by rejecting when an independent uniform random variable $U$ on $(0,1)$ satisfies $U<\varphi (x)$. A deterministic test has only the values zero and one. Under a simple hypothesis $H_j$, its rejection probability is ${\mathbb{E}}_j\varphi (X)$, which is its test size for $j=0$ and its statistical power for $j=1$.

The Neyman-Pearson lemma permits randomization on an equality set of the form $f_1(x)=k{f}_0(x)$ to reach a prescribed test size. Continuous observations do not rule out an equality set of positive probability: if both hypotheses give the same probability density function, their likelihood ratio is identically one. This explains why a precise statement of the lemma includes boundary randomization even for continuous data.

The expected rate of detections exceeding a stated threshold under a specified background-only model. A time rate and a per-trial significance level require different normalizations.

A statistical hypothesis is a specified set of possible data-generating probability distributions, usually expressed as a restriction on model parameters. A simple statistical hypothesis specifies one distribution; a composite statistical hypothesis permits several. A hypothesis test compares a null hypothesis with an alternative hypothesis using a statistic calibrated under the null.

A sequential probability ratio test accumulates the log-likelihood ratio for two specified simple statistical hypotheses until it reaches an upper or lower boundary. An upper crossing favors the alternative, a lower crossing favors the null, and observations continue between them. Error-calibrated boundaries are approximately $\log [(1-\beta )/\alpha ]$ and $\log [\beta /(1-\alpha )]$; discrete overshoot means these approximations need calibration.

A result is statistically significant at a specified significance level $\alpha$ when its valid p-value is at most $\alpha$ and the corresponding null hypothesis is rejected. Significance does not measure effect size, establish practical importance, or give the probability that the null is true. Software stars encode thresholds for $p$-values; they do not supply an additional statistic.

For a known background rate $b$ observed for exposure $T$, independent events give a Poisson distribution with mean ${\mu }_b=bT$. An excess-count test of the background-only null hypothesis uses the upper-tail P-value in the title. The tail, rather than the probability of exactly $n$, counts all outcomes at least as discrepant in the chosen direction.

This probability is conditional on the null model; it is not the posterior probability that the null model is true. A stipulated discovery threshold is met only when the tail is below that threshold. Uncertainty in the background rate or selecting among many searches needs additional modelling rather than being absorbed into the same known-mean formula.

Searching many possible signal locations increases the chance of finding an apparently unusual fluctuation somewhere. A local p-value for a specified location differs from the global p-value for the search over locations.^[ref]

A two-sided test detects departures in either direction from the null hypothesis, for example $\theta \ne {\theta }_0$. With a continuous symmetric null statistic centred at zero, its p-value is $P_0(|T|\ge |t_{\mathrm{obs}}|)=2P_0(T\ge |t_{\mathrm{obs}}|)$. This is the usual coefficient test printed by Gaussian regression summaries, unlike a directional one-sided comparison.

A one-sided test targets deviations in one specified direction, such as $\theta >{\theta }_0$. If larger values of a statistic $T$ indicate this direction, its p-value is an upper-tail null probability $P_0(T\ge t_{\mathrm{obs}})$. The direction should be chosen from the scientific question before examining the data. An upper normal test uses $1-\Phi (z)$; a lower test uses $\Phi (z)$.

The alternative hypothesis specifies the parameter values or distributions against which a null hypothesis is tested. If the null is $\theta \in {\Theta }_0$, the alternative is $\theta \in {\Theta }_1$ with ${\Theta }_0\cap {\Theta }_1=\varnothing$. For example an upper one-sided comparison uses $\theta >{\theta }_0$, while a two-sided comparison uses $\theta \ne {\theta }_0$. The alternative determines the relevant rejection tail. A p-value is computed under the null; it is not the probability that the alternative is false.

For independent multivariate normal observations with positive-definite covariance, this statistic is the maximum squared studentized mean contrast. With unbiased sample covariance matrix $S$ and $n>p$, its null distribution obeys $(n-p)T^2/(p(n-1))\sim F_{p,n-p}$. This follows from independence of the Gaussian sample mean and the residual Wishart distribution.

Under a nonsingular affine transformation $Y=AX+b$, transform the null mean by the same map. Then the centered sample mean becomes $A(\bar{X}-{\mu }_0)$ and the sample covariance matrix becomes $AS{A}^T$. Since $(AS{A}^T{)}^{-1}=A^{-T}{S}^{-1}{A}^{-1}$, the quadratic form in Hotelling's T-squared statistic is unchanged. Its value and test decision are consequently invariant under changes of units, origin and nonsingular coordinates.

For a fixed rank-$m$ constraint matrix $B$, apply Hotelling's T-squared statistic to the transformed observations $B{X}_j$. Under $B\mu =0$, $(n-m)T_B^2/(m(n-1))\sim F_{m,n-m}$ for $n>m$. If the displayed contrasts are dependent, use a basis of their row space and replace $m$ by the rank.

At equally spaced distinct times, a mean profile is affine exactly when its consecutive second differences vanish. These form $p-2$ independent contrasts. At unequal times, compare adjacent slopes instead: $({\mu }_{i+1}-{\mu }_i)/(t_{i+1}-t_i)-({\mu }_i-{\mu }_{i-1})/(t_i-t_{i-1})$. A Hotelling test of linear hypotheses tests these restrictions jointly with unknown covariance.

A rejection region is the set of sample outcomes on which a hypothesis test rejects its null hypothesis. Its probability under the null determines the type I error rate.

For paired binary observations $(C_i,M_i)$, let $b$ count pairs $(1,0)$ and $c$ pairs $(0,1)$. Under the null hypothesis of equal marginal event probabilities, these discordant outcomes have equal probabilities. Conditional on $b+c$, $b$ has the binomial distribution $\mathrm{Bin}(b+c,1/2)$, giving an exact test. For enough discordant pairs, $(b-c)/\sqrt{b+c}$ has an approximate standard normal distribution, or its square has an approximate one-degree-of-freedom chi-squared distribution. Concordant pairs provide no information about the direction of the marginal difference. If there are no discordant pairs, there is no directional information and the divided statistic is not computed. The pairing and independence between pairs are essential; the usual independent-two-sample formula is not a paired test.

Let a paired binary table have cells $a,b,c,d$ and $n=a+b+c+d$. The uncorrected McNemar test is $(b-c{)}^2/(b+c)$, while the ordinary independent-sample chi-squared test on the two marginal allele columns is $2n(b-c{)}^2/[(2a+b+c)(2d+b+c)]$. For nonzero difference and positive denominators, they coincide exactly when $(2a+b+c)(2d+b+c)=2n(b+c)$. Such numerical coincidence does not justify discarding pairing; altering concordant counts can change the unpaired statistic while leaving McNemar's statistic unchanged.

// Target: biology.bigb

For a paired binary difference $D=C-M$, set $\delta =P(C=1)-P(M=1)$ and $q=P(C\ne M)$. Then $\mathbb{E}D=\delta$ and $\mathrm{Var}(D)=q-{\delta }^2$. Under equal marginal probabilities the variance is $q$. The McNemar test has approximate upper rejection boundary $z_{1-\alpha /2}\sqrt{nq}$ for the sum of differences. Requiring that the alternative mean $n\delta$ exceed this boundary by $z_{1-\beta }\sqrt{n(q-{\delta }^2)}$ gives the displayed planning size for a positive alternative, neglecting its very small lower-tail rejection probability. Use $z_{1-\alpha }$ for a pre-specified one-sided test. The marginal probabilities alone do not determine $q$: co-occurrence of positive outcomes must be specified or bounded. Discreteness and clustered data can change achieved statistical power.

For independent effect estimates $1.01$ with standard error $0.50$, and $0.99$ with standard error $0.51$, one two-sided Wald test is significant at five percent and the other is not. But their difference is $0.02$ with standard error $\sqrt{0.50^2+0.51^2}\approx 0.714$, giving a negligible test statistic. Hence different significance labels do not establish an interaction or between-study heterogeneity; compare the effects directly.^[ref]

Generate independent replicated datasets under a fully specified null hypothesis and calculate the same predictive discrepancy statistic for them and the observation. Their null exchangeability gives a finite-simulation rank test. The displayed add-one tail probability is valid with conservative treatment of ties, and avoids reporting zero solely because no replicate exceeds the observation.

A measurable summary of a possible dataset is chosen to detect disagreement with a specified predictive model. Compare its observed value with its distribution under replicated datasets. Large-discrepancy statistics can be checked with $\mathbb{P}(T(X^{\mathrm{rep}})\ge T(x))$. A posterior predictive check integrates over posterior parameter uncertainty; a fully specified null instead supplies a fixed reference law.

A scan statistic searches a family of candidate locations or subsets and reports the largest local statistic. Under a null hypothesis, a union bound converts a tail estimate for each candidate into a bound for the maximum; no independence between local statistics is required. Structured alternatives often permit a chi-squared testing lower bound using the overlap geometry of two candidates.

For independent uniformly placed cyclic intervals of length $k<d$, an overlap is possible at at most $2k-1$ starting-point offsets, or at all $d$ offsets if that number exceeds $d$. Thus the overlap probability is at most $\min \{1,(2k-1)/d\}$. If $R=|S\cap T|$, then $0\le R\le k$ and $\mathbb{E}(e^{aR/k}-1)\le (2k/d)(e^a-1)$ for $a\ge 0$. The overlap law is not the hypergeometric distribution for unrestricted random subsets.

A quadratic scan statistic maximizes a directional empirical second moment over candidate unit vectors. For a Gaussian random vector sample with a rank-one covariance spike in one of those directions, the matching statistic has its scale multiplied by the spike's eigenvalue. A chi-squared concentration inequality controls each direction.

A least-favourable configuration maximizes a test procedure’s type I error probability over allowed null hypothesis parameters. For one-sided normal comparisons with a fixed covariance matrix, coupling by common centered variables shows monotonicity in the means. For the familywise error rate, mixed true and false null configurations must also be checked.

The Pearson chi-squared test of homogeneity tests whether several independent categorical samples have the same category probabilities. Under the null, expected cell counts equal the product of the corresponding row and column totals divided by the total sample size. The Pearson chi-squared statistic for contingency tables $\sum (O-E{)}^2/E$ has an asymptotic chi-squared distribution with $(r-1)(c-1)$ degrees of freedom, when expected counts are sufficiently large. The sampling design differs from a Pearson chi-squared test of independence, though the statistic has the same form.

For independent $R_i\sim \mathrm{Bin}(n_i,p_i)$, a group with $n_i=0$ necessarily has $R_i=0$ and contributes likelihood one. Its probability parameter is unidentifiable and supplies no restriction to test. If $k$ groups have positive sample sizes, the unrestricted model has $k$ probability parameters and the common-probability model has one; the asymptotic homogeneity test therefore has $k-1$ degrees of freedom. Zero expected-count rows must be removed rather than divided by in the Pearson statistic.

For observed counts $O_{ij}$ and positive expected counts $E_{ij}$ in a contingency table, the statistic is $X^2={\sum }_{ij}(O_{ij}-E_{ij}{)}^2/E_{ij}$. Under common category probabilities or statistical independence, estimate $E_{ij}$ by the product of the row and column totals divided by the grand total. With large expected counts, this gives the test statistic for the Pearson chi-squared test of homogeneity or the Pearson chi-squared test of independence, with asymptotic chi-squared distribution on $(r-1)(c-1)$ degrees of freedom.

Consider independent binomial counts $X_i\sim \mathrm{Bin}(n_i,p)$ under the common-probability null hypothesis, with fixed positive row sizes $n_i$. Let $N={\sum }_i{n}_i$, $K={\sum }_i{X}_i$ and assume $0<K<N$. Conditional on $K$, the probability mass function is the displayed multivariate hypergeometric distribution: in the joint likelihood, the factor $p^K(1-p{)}^{N-K}$ is common to all admissible tables and cancels. The Vandermonde identity gives the denominator, so the conditional distribution is free of the unknown nuisance parameter $p$.

For a table $x$, its Pearson chi-squared statistic isAn exact conditional p-value sums the displayed probabilities over all tables with the same margins and $Q(x)\ge Q(x_{\mathrm{obs}})$. The ordering is by the Pearson chi-squared statistic, rather than by the probability of an individual table; different exact-test orderings can give different answers. Enumeration is practical when $K$ and the number of rows are small. Monte Carlo tests can approximate the same tail by sampling tables with fixed margins.^[ref]

An exact arithmetic comparison is easy too. Let $M$ be the least common multiple of the row sizes. Compare the integersince every table has the same positive denominator $MK(N-K)$. This includes boundary ties without a floating-point tolerance. Conditional p-values are discrete and can be conservative; “exact” concerns their null calibration, not equality of rejection probability with every chosen significance level.

For a two-by-two contingency table, subtracting one half from each absolute observed-minus-expected count difference, with truncation at zero, gives the displayed corrected Pearson chi-squared statistic. Expected counts use the independence model $E_{ij}=O_{i+}{O}_{+j}/N$. The subtraction compensates approximately for using a continuous chi-squared distribution for discrete counts. It can make the test conservative. The corrected statistic should be distinguished from the ordinary uncorrected Pearson chi-squared statistic.

A statistical test is a measurable rule for deciding whether to reject a null hypothesis. Allowing randomisation, it is represented by a function $\phi (x)\in [0,1]$ giving the conditional rejection probability after observing $x$. A deterministic test takes only values zero and one. Under a simple null, $E_0\phi$ is its size of a statistical test; under an alternative, $E_1\phi$ is its statistical power. Randomisation on a likelihood ratio equality set allows a Neyman-Pearson lemma threshold test to attain a prescribed size even for discrete data.

At level $\alpha$, a most powerful test against a specified simple alternative maximises statistical power among statistical tests whose size of a statistical test is at most $\alpha$. The Neyman-Pearson lemma constructs it by a likelihood ratio threshold. A uniformly most powerful test is most powerful simultaneously against every member of the specified alternative family; maximising power at one alternative alone need not give uniform optimality.

A test statistic is a measurable function of the sample whose observed value is compared with a null distribution or resampling distribution to decide whether to reject a hypothesis.

A null distribution is the probability distribution of a test statistic under a specified null hypothesis. It calibrates a hypothesis test's test size and p-value. For a simple hypothesis it is one distribution; for a composite null hypothesis it may depend on unknown nuisance parameters.

Student's t-test divides a normally distributed estimated effect by its estimated standard error. Under its null hypothesis the statistic has a Student's t-distribution, with degrees of freedom determined by the variance estimate.

For one two-sided t procedure with observed estimate $\hat{\Delta }$, positive standard error $s$, null value ${\Delta }_0$ and reference Student t-distribution with $\nu$ statistical degrees of freedom, the test statistic and p-value must satisfyIts symmetric confidence interval at coverage probability $1-\alpha$ is $[\hat{\Delta }-t_{\nu ,1-\alpha /2}s,\hat{\Delta }+t_{\nu ,1-\alpha /2}s]$ and therefore has midpoint $\hat{\Delta }$. When the same standard error and reference distribution are used, excluding ${\Delta }_0$ is equivalent to $|t|>t_{\nu ,1-\alpha /2}$ and hence $p<\alpha$ for a continuous reference distribution. This proves the test–interval equivalence and supplies several independent checks on a printed output. An approximate Welch's t-test uses its same approximate statistical degrees of freedom throughout. Rounding can blur a boundary decision but cannot explain a confidence-interval midpoint far from the reported estimate.

A one-sample Student t-test on independent pairs' differences $d_i$. It tests a specified mean difference $d_0$, using the standard deviation of those differences rather than treating both measurements in a pair as independent. Exact small-sample inference assumes normally distributed differences.

For two independent samples from approximately normal distributions, Welch's t-test compares their means without assuming equal unknown variances. The displayed test statistic uses sample variances $s_X^2,s_Y^2$. Under the equal-mean null hypothesis, its reference Student t-distribution has approximate statistical degrees of freedomA p-value from this approximation addresses the specified statistical error model; it cannot distinguish a material effect from an unmeasured experimenter bias when sample and experimenter are confounded.

For two samples of sizes $n_1,n_2>1$, write $a=s_1^2/n_1$ and $b=s_2^2/n_2$, with $a+b>0$. The reference statistical degrees of freedom in Welch's t-test satisfyIndeed the Cauchy-Schwarz inequality applied to $(a/\sqrt{n_1-1},b/\sqrt{n_2-1})$ and $(\sqrt{n_1-1},\sqrt{n_2-1})$ gives the inequality after division. Thus an independent two-sample Welch's t-test on $N$ complete observations cannot have reference statistical degrees of freedom exceeding $N-2$. The exact pooled two-sample t statistic also uses $N-2$, under its stronger equal-variance assumptions.

For independent normal samples of sizes $m,n$ with a common unknown variance and equal means under the null, $Z=(\overline{X}-\overline{Y})/(\sigma \sqrt{1/m+1/n})$ is standard normal. The independent centered sample sums of squares give $V=(m+n-2)S_p^2/{\sigma }^2\sim {\chi }_{m+n-2}^2$, independently of $Z$. Thus $Z/\sqrt{V/(m+n-2)}$ has the exact Student t-distribution, giving the statistic $(\overline{X}-\overline{Y})/(S_p\sqrt{1/m+1/n})$. A lower one-sided test uses its lower-tail probability.

For an independent sample from a normal distribution, maximization of the likelihood over the unknown variance gives the likelihood ratio for a fixed mean against an unrestricted mean as displayed, where $t=\sqrt{n}(\bar{X}-{\mu }_0)/S$ and $S^2={\sum }_i(X_i-\bar{X}{)}^2/(n-1)$. It decreases with $|t|$. Under the null, the normal sample mean is independent of its centered sum of squares, so $t$ has the exact Student t-distribution with $n-1$ degrees of freedom. The resulting two-sided test requires no asymptotic chi-squared approximation.

In a fixed full-rank normal linear model, the studentized estimator of one coefficient has a noncentral t-distribution with noncentrality equal to that coefficient divided by its standard deviation. The fitted and residual orthogonal projections of the Gaussian noise are independent. Therefore the statistical power of its exact Student's t-test does not depend on the other regression coefficients, with design and variance fixed.

A fixed alternative keeps a non-null data-generating distribution unchanged as sample size grows. A consistent test rejects with probability tending to one under every fixed alternative in its stated class.

A local alternative approaches the null as sample size grows, commonly at distance $n^{-1/2}$. It reveals the nontrivial limiting power of a test near the null.

A randomization test compares a statistic with values obtained by applying transformations that preserve the joint distribution under the null hypothesis.

A conditional randomization test resamples a tested variable from its known or estimated conditional distribution given covariates. Under conditional independence from the outcome, the observed and resampled test statistics are exchangeable conditional on those covariates and outcomes.

A permutation test uses relabelings of observations as its null-preserving transformations and computes a p-value from the resulting orbit of the test statistic.

A sign-flip randomization test independently multiplies observations by signs in $\{-1,1\}$. It is finite-sample exact for a zero-centered null when the joint distribution is invariant under those sign changes.

The null hypothesis is the set of parameter values or probability distributions against which a statistical test controls its probability of rejection.

A simple hypothesis specifies one probability distribution completely. A composite hypothesis permits more than one distribution or parameter value.

The size of a test is the supremum, over distributions in its null hypothesis, of its rejection probability. For a simple null it is just the null rejection probability; a level-$\alpha$ test has size at most $\alpha$.

The significance level is the chosen upper bound on a test's probability of rejecting the null hypothesis when it is true.

A result is statistically significant when its calibrated p-value falls below this chosen threshold.

A p-value is the probability, computed under a null hypothesis, of obtaining a test statistic at least as unfavorable to that hypothesis as the observed value.

A random variable $P\in [0,1]$ is super-uniform when $\mathbb{P}(P\le \alpha )\le \alpha$ for every $\alpha \in [0,1]$. A valid p-value is super-uniform under its null hypothesis.

Fisher's exact test conditions on the margins of a two-by-two contingency table and compares one cell count with its hypergeometric null distribution.

For two independent binomial counts, form the Fisher exact test rejection region from their conditional null hypergeometric distribution. Summing alternative joint binomial probabilities over that region gives its exact power. For equal group sizes, the null allocation is symmetric, so the two-sided probability-ordering test agrees with doubled lower-tail probabilities. Conservative test discreteness can make a normal-approximation sample size underpowered even when both samples are large but event counts are low.

A maximin test maximizes the smallest power over a composite alternative subject to controlling the largest rejection probability over a composite null.

Multiple hypothesis testing applies tests to a family of null hypotheses while controlling an aggregate error criterion.

For a nonempty finite index set $I$, this hypothesis asserts that every member $H_i$ is true simultaneously. A valid local test must have its stated size under every parameter satisfying the entire intersection. The Bonferroni correction within $I$ always supplies such a test from valid elementary p-values.

The closure is the collection of all nonempty intersection hypotheses from the original elementary family, including singletons. It is not a topological closure. The closed testing procedure requires local level-$\alpha$ tests for these intersections and rejects an elementary hypothesis only when all intersections containing it reject.

Order $m$ p-values and reject the first $r$ hypotheses, where $r=\max \{j:p_{(j)}\le \alpha /(m-j+1)\}$, with $r=0$ for an empty set. Under independent valid true-null p-values, this step-up procedure controls the familywise error rate. One proof embeds its rejections in the closed testing procedure with Simes tests: every intersection containing a rejected hypothesis passes its local Simes test, so closed-testing control of the familywise error rate applies.

For $m\ge 1$ independent valid null p-values, the displayed inequality follows by applying the Benjamini-Hochberg procedure with all nulls true. Its false discovery rate then equals the probability of at least one rejection. Valid means $\mathrm{Pr}(p_i\le t)\le t$ for $0\le t\le 1$; independence is a sufficient hypothesis, and arbitrary dependence is not covered by this proof.

For the intersection of $m$ null hypotheses, reject when ${\min }_{i}m{p}_{(i)}/i\le \alpha$. The Simes inequality establishes a level-$\alpha$ test under independence of valid true-null p-values. The displayed quantity is its combined p-value.

Given valid p-values $q_J$ for constancy on intervals of length at least two, reject $I$ when ${\max }_{J\supseteq I}n{q}_J/|J|\le \alpha$. Every rejected true interval forces rejection of its maximal constant block. A union bound over the non-singleton maximal blocks gives familywise error rate at most $\alpha$, since their lengths sum to at most $n$. Singleton blocks have no tested interval in this formulation.

For a null hypothesis that is a union $H_0={\bigcup }_{i\in I}{H}_i$, an intersection-union test rejects $H_0$ only when every component $H_i$ is rejected. If each component test has level $\alpha$, then under any point of $H_0$ at least one component null is true, so the combined test also has level at most $\alpha$.

The Bonferroni correction tests each of $m$ hypotheses at level $\alpha /m$. The union bound then controls the familywise error rate by $\alpha$ under arbitrary dependence.

For positive deterministic weights $w_i$, the weighted Bonferroni correction rejects $H_i$ whenThe union bound over the true nulls controls the familywise error rate without an independence assumption.

Order $q_i=p_i/w_i$ increasingly. At rank $k$, reject the corresponding hypothesis whenand otherwise stop. The first ordered true null reduces FWER control to a weighted union bound; the shrinking denominator makes this procedure at least as powerful as the Weighted Bonferroni correction.

The Holm–Bonferroni method orders p-values and compares the $j$th smallest with $\alpha /(m-j+1)$, stopping at the first failed comparison. It controls the familywise error rate under arbitrary dependence.

If the Holm step-down procedure makes a false rejection, it must reject the first true null in the ordered list. All $m_0$ true nulls remain at that step, so its cutoff is at most $\alpha /m_0$. The union bound on their marginally valid p-values gives familywise error rate at most $\alpha$. This proves strong control under arbitrary dependence, and super-uniformity suffices.

Order $m$ p-values as $p_{(1)}\le \cdots \le p_{(m)}$ and take the largest $k$ with $p_{(k)}\le \alpha k/m$. The Benjamini-Hochberg procedure rejects the hypotheses corresponding to $p_{(1)},\dots ,p_{(k)}$ and controls the false discovery rate under independence and several forms of positive dependence.

Let $R$ be the number of rejections by the Benjamini-Hochberg procedure at level $\alpha$, and let $R_i$ be the rejection count after replacing $p_i$ by zero. Then $R_i\ge 1$ andIf $i$ was rejected, lowering its p-value does not change any ordered p-value beyond rank $R$, so $R_i=R$. Conversely, if $p_i\le \alpha R_i/m$, the other $R_i-1$ rejected values together with $p_i$ satisfy the original threshold; monotonicity gives $R=R_i$. For a valid true-null p-value independent of the others, conditioning on $R_i$ bounds the expectation of the right side by $\alpha /m$. Summing over true nulls proves false discovery rate at most $m_0\alpha /m$.

Replace two p-values $P_i,P_j$ by zeros in the Benjamini-Hochberg procedure at level $\alpha$, keeping the original denominator $m$, and let $R^{(ij)}$ be the resulting rejection count. Equivalently, remove the two values and apply the step-up critical values $\alpha (k+2)/m$ to the remaining ordered p-values; their rejection count is $R_{-ij}=R^{(ij)}-2$. For $r\ge 2$,If both original values are rejected, lowering them changes no ordered rank greater than $r$, so the rejection count stays $r$. Conversely, if the modified count is $r$ and both original values meet the rank-$r$ threshold, reinserting them leaves at least $r$ qualifying values. Monotonicity under lowering gives at most $r$ original rejections. This identity is deterministic and does not require independence or a distributional assumption.

For independent p-values, with $m_0$ true-null values independent and uniform on $[0,1]$, let $R^{(i)}$ denote the Benjamini-Hochberg procedure rejection count after replacing one true-null value by zero. The Benjamini-Hochberg leave-one-out identity givesFor distinct true-null indices, the Benjamini-Hochberg leave-two-out identity givesIndeed, conditional on the remaining values and $R^{(ij)}=r$, the two independent uniform values both meet their threshold with probability $(\alpha r/m{)}^2$, cancelling the denominator $r^2$. Expanding the square of the false rejection count into its diagonal terms and ordered distinct pairs, and using symmetry among the true-null values, provesIf no null is true, both sides are zero; if exactly one is true, the pair contribution is absent. This formula complements the exact false discovery rate $\alpha m_0/m$ by describing second-moment variability.

For the Benjamini-Hochberg procedure, replace a true-null p-value $P_i$ by zero and call the new rejection count $R^{(i)}$. The Benjamini-Hochberg leave-one-out identity identifies rejection with $R=r$ with the event $P_i\le \alpha r/m$ and $R^{(i)}=r$. If $P_i$ is uniform and independent of all the other p-values, each possible rejection count contributes $\alpha \mathbb{P}(R^{(i)}=r)/m$ to its expected false-discovery fraction. Summation gives $\alpha /m$ per true null and the displayed exact false discovery rate. Super-uniform independent nulls yield the corresponding upper bound.

The false discovery rate is $\mathbb{E}[V/(R\vee 1)]$, where $V$ is the number of rejected true null hypotheses and $R$ is the total number of rejections.

In multiple hypothesis testing, let $R$ count all rejections and $V$ count rejected true null hypotheses. The false discovery proportion is the random variable $V/(R\vee 1)$, defined to be zero when $R=0$. Its expectation is the false discovery rate. Distinguishing the realized proportion from its expectation matters: controlling the false discovery rate does not by itself give a high-probability bound on the realized proportion.

The familywise error rate is the probability of making at least one false rejection in a family of hypothesis tests.

For nonempty index sets forming a laminar family of sets, reject an intersection hypothesis when the displayed adjusted value is at most $\alpha$. The inclusion-maximal true sets are disjoint and have total size at most $m$. Any false rejection forces some maximal true $J$ to have $p_J\le \alpha |J|/m$. Validity and the union bound control the familywise error rate by $\alpha$, without independence or a complete closure.

Choose a level-$\alpha$ local test for every nonempty intersection $H_I={\bigcap }_{i\in I}{H}_i$. The closed testing procedure rejects an elementary hypothesis $H_i$ exactly when every intersection hypothesis containing it is rejected by its local test.

If any true elementary hypothesis is rejected by a closed testing procedure, then the intersection of all true null hypotheses must also be rejected. Its local test has level $\alpha$, so the familywise error rate is at most $\alpha$ under arbitrary dependence.

A joint hypothesis test imposes several parameter restrictions simultaneously and accounts for dependence among their estimators; separate one-parameter confidence intervals do not generally determine its result.

For a test with rejection region $R$, its power function isIts supremum over the null parameter space is the size of the test.

Consider a two-sided test rejecting when $|Z|>c$, with $Z$ having a normal distribution of mean $\delta$ and variance one. Here $\Phi$ is the cumulative distribution function of the standard normal distribution. Under a nonzero alternative $\delta$, the type II error probability is the displayed function. It tends to the nonrejection probability under the null as $\delta \to 0$, and to zero as $\delta \to +\infty$. A particular observed $Z=z$ has positive density under all these alternatives. Its null-calibrated p-value therefore does not select one value of $\beta$. The power function describes repeated-sampling performance at specified alternatives.

A level-$\alpha$ test is uniformly most powerful when its power is at least that of every other level-$\alpha$ test at every parameter value in the alternative.

For a Poisson sample tested against mean ${\mu }_0=1$, the exact uniformly most powerful one-sided test thresholds the sample sum and randomizes on its boundary if necessary. A normal approximation gives critical sum $n+z_{1-\alpha }\sqrt{n}$. Equating its standardized alternative threshold to $-z_{1-\beta }$ gives the displayed design for approximate power $1-\beta$ at mean ${\mu }_1>1$. At size $0.05$ and target power $0.9$, the two quantiles are $1.645$ and $1.282$. Rounding up supplies an integer design; exact Poisson power still requires discrete evaluation.

If two tests have the same size and each has strictly greater power than the other at some alternative, neither test is uniformly most powerful.

The Pearson chi-squared test compares observed categorical counts with null expected counts using the displayed statistic. Its applications include Pearson chi-squared goodness-of-fit test for specified probabilities and Pearson chi-squared test of independence for contingency tables; the fitted restrictions determine the limiting degrees of freedom.

For $K$ independent strata, each containing an $r\times c$ contingency table, test independence of the two table variables within every stratum. Under adequate expected counts and the usual multinomial distribution sampling assumptions, each stratum's Pearson chi-squared statistic has asymptotic chi-squared distribution with $(r-1)(c-1)$ statistical degrees of freedom. Their sum has asymptotic chi-squared distribution with $K(r-1)(c-1)$ statistical degrees of freedom, by the additivity of independent chi-squared distributions. This is a test of conditional independence, not of equality of a common effect across strata; the latter needs an interaction contrast or another specified homogeneity test.

For observed cell counts $O_j$ and null expected counts $E_j$, the statistic isWith $k$ specified null cell probabilities and sufficiently large expected counts, its null limit is ${\chi }_{k-1}^2$.

For $N$ independent groups each containing $m$ independent Bernoulli trials with common probability $\theta$, the group count has a binomial distribution. If $O_j$ groups have count $j$, the maximum-likelihood estimate is $\hat{\theta }={\sum }_{j}j{O}_j/(mN)$. Form the Pearson chi-squared statistic using $E_j=N\left(\genfrac{}{}{0ex}{}{m}{j}\right){\hat{\theta }}^j(1-\hat{\theta }{)}^{m-j}$. With all $m+1$ categories retained, an interior true parameter, and sufficiently large expected counts, its null limit is a chi-squared distribution with $m-1$ statistical degrees of freedom: one degree is removed by the fixed total and another by fitting $\theta$. Small expected counts require care with this approximation. If categories are pooled, the fitted parameter and calibration should correspond to the pooled statistical model; simply pooling a statistic after fitting the unpooled model does not automatically give the usual chi-squared limit.

For an $r$ by $c$ contingency table, the statistichas asymptotically a chi-squared distribution with $(r-1)(c-1)$ degrees of freedom under independence.