Statistical hypothesis testing

Statistical hypothesis testing compares data against a null hypothesis using a controlled rejection probability.

For independent effect estimates $1.01$ with standard error $0.50$, and $0.99$ with standard error $0.51$, one two-sided Wald test is significant at five percent and the other is not. But their difference is $0.02$ with standard error $\sqrt{0.50^2+0.51^2}\approx 0.714$, giving a negligible test statistic. Hence different significance labels do not establish an interaction or between-study heterogeneity; compare the effects directly.

Generate independent replicated datasets under a fully specified null hypothesis and calculate the same predictive discrepancy statistic for them and the observation. Their null exchangeability gives a finite-simulation rank test. The displayed add-one tail probability is valid with conservative treatment of ties, and avoids reporting zero solely because no replicate exceeds the observation.

A measurable summary of a possible dataset is chosen to detect disagreement with a specified predictive model. Compare its observed value with its distribution under replicated datasets. Large-discrepancy statistics can be checked with $\mathbb{P}(T(X^{\mathrm{rep}})\ge T(x))$. A posterior predictive check integrates over posterior parameter uncertainty; a fully specified null instead supplies a fixed reference law.

A scan statistic searches a family of candidate locations or subsets and reports the largest local statistic. Under a null hypothesis, a union bound converts a tail estimate for each candidate into a bound for the maximum; no independence between local statistics is required. Structured alternatives often permit a chi-squared testing lower bound using the overlap geometry of two candidates.

For independent uniformly placed cyclic intervals of length $k<d$, an overlap is possible at at most $2k-1$ starting-point offsets, or at all $d$ offsets if that number exceeds $d$. Thus the overlap probability is at most $\min \{1,(2k-1)/d\}$. If $R=|S\cap T|$, then $0\le R\le k$ and $\mathbb{E}(e^{aR/k}-1)\le (2k/d)(e^a-1)$ for $a\ge 0$. The overlap law is not the hypergeometric distribution for unrestricted random subsets.

A quadratic scan statistic maximizes a directional empirical second moment over candidate unit vectors. For a Gaussian random vector sample with a rank-one covariance spike in one of those directions, the matching statistic has its scale multiplied by the spike's eigenvalue. A chi-squared concentration inequality controls each direction.

A least-favourable configuration maximizes a test procedure’s Type I error probability over allowed null hypothesis parameters. For one-sided normal comparisons with a fixed covariance matrix, coupling by common centered variables shows monotonicity in the means. For the familywise error rate, mixed true and false null configurations must also be checked.

The Pearson chi-squared test of homogeneity tests whether several independent categorical samples have the same category probabilities. Under the null, expected cell counts equal the product of the corresponding row and column totals divided by the total sample size. The Pearson chi-squared statistic for contingency tables $\sum (O-E{)}^2/E$ has an asymptotic chi-squared distribution with $(r-1)(c-1)$ degrees of freedom, when expected counts are sufficiently large. The sampling design differs from a Pearson chi-squared test of independence, though the statistic has the same form.

For observed counts $O_{ij}$ and positive expected counts $E_{ij}$ in a contingency table, the statistic is $X^2={\sum }_{ij}(O_{ij}-E_{ij}{)}^2/E_{ij}$. Under common category probabilities or statistical independence, estimate $E_{ij}$ by the product of the row and column totals divided by the grand total. With large expected counts, this gives the test statistic for the Pearson chi-squared test of homogeneity or the Pearson chi-squared test of independence, with asymptotic chi-squared distribution on $(r-1)(c-1)$ degrees of freedom.

A statistical test is a measurable rule for deciding whether to reject a null hypothesis. Allowing randomisation, it is represented by a function $\phi (x)\in [0,1]$ giving the conditional rejection probability after observing $x$. A deterministic test takes only values zero and one. Under a simple null, $E_0\phi$ is its size of a statistical test; under an alternative, $E_1\phi$ is its statistical power. Randomisation on a likelihood ratio equality set allows a Neyman-Pearson lemma threshold test to attain a prescribed size even for discrete data.

At level $\alpha$, a most powerful test against a specified simple alternative maximises statistical power among statistical tests whose size of a statistical test is at most $\alpha$. The Neyman-Pearson lemma constructs it by a likelihood ratio threshold. A uniformly most powerful test is most powerful simultaneously against every member of the specified alternative family; maximising power at one alternative alone need not give uniform optimality.

A test statistic is a measurable function of the sample whose observed value is compared with a null distribution or resampling distribution to decide whether to reject a hypothesis.

Student's t-test divides a normally distributed estimated effect by its estimated standard error. Under its null hypothesis the statistic has a Student's t-distribution, with degrees of freedom determined by the variance estimate.

In a fixed full-rank normal linear model, the studentized estimator of one coefficient has a noncentral t-distribution with noncentrality equal to that coefficient divided by its standard deviation. The fitted and residual orthogonal projections of the Gaussian noise are independent. Therefore the statistical power of its exact Student's t-test does not depend on the other regression coefficients, with design and variance fixed.

A fixed alternative keeps a non-null data-generating distribution unchanged as sample size grows. A consistent test rejects with probability tending to one under every fixed alternative in its stated class.

A local alternative approaches the null as sample size grows, commonly at distance $n^{-1/2}$. It reveals the nontrivial limiting power of a test near the null.

A randomization test compares a statistic with values obtained by applying transformations that preserve the joint distribution under the null hypothesis.

A conditional randomization test resamples a tested variable from its known or estimated conditional distribution given covariates. Under conditional independence from the outcome, the observed and resampled test statistics are exchangeable conditional on those covariates and outcomes.

A permutation test uses relabelings of observations as its null-preserving transformations and computes a p-value from the resulting orbit of the test statistic.

A sign-flip randomization test independently multiplies observations by signs in $\{-1,1\}$. It is finite-sample exact for a zero-centered null when the joint distribution is invariant under those sign changes.

The null hypothesis is the set of parameter values or probability distributions against which a statistical test controls its probability of rejection.

A simple hypothesis specifies one probability distribution completely. A composite hypothesis permits more than one distribution or parameter value.

The size of a test is the supremum, over distributions in its null hypothesis, of its rejection probability. For a simple null it is just the null rejection probability; a level-$\alpha$ test has size at most $\alpha$.

The significance level is the chosen upper bound on a test's probability of rejecting the null hypothesis when it is true.

A p-value is the probability, computed under a null hypothesis, of obtaining a test statistic at least as unfavorable to that hypothesis as the observed value.

A random variable $P\in [0,1]$ is super-uniform when $\mathbb{P}(P\le \alpha )\le \alpha$ for every $\alpha \in [0,1]$. A valid p-value is super-uniform under its null hypothesis.

Fisher's exact test conditions on the margins of a two-by-two contingency table and compares one cell count with its hypergeometric null distribution.

For two independent binomial counts, form the Fisher exact test rejection region from their conditional null hypergeometric distribution. Summing alternative joint binomial probabilities over that region gives its exact power. For equal group sizes, the null allocation is symmetric, so the two-sided probability-ordering test agrees with doubled lower-tail probabilities. Conservative test discreteness can make a normal-approximation sample size underpowered even when both samples are large but event counts are low.

A maximin test maximizes the smallest power over a composite alternative subject to controlling the largest rejection probability over a composite null.

Multiple hypothesis testing applies tests to a family of null hypotheses while controlling an aggregate error criterion.

For a nonempty finite index set $I$, this hypothesis asserts that every member $H_i$ is true simultaneously. A valid local test must have its stated size under every parameter satisfying the entire intersection. The Bonferroni correction within $I$ always supplies such a test from valid elementary p-values.

The closure is the collection of all nonempty intersection hypotheses from the original elementary family, including singletons. It is not a topological closure. The closed testing procedure requires local level-$\alpha$ tests for these intersections and rejects an elementary hypothesis only when all intersections containing it reject.

Order $m$ p-values and reject the first $r$ hypotheses, where $r=\max \{j:p_{(j)}\le \alpha /(m-j+1)\}$, with $r=0$ for an empty set. Under independent valid true-null p-values, this step-up procedure controls the familywise error rate. One proof embeds its rejections in the closed testing procedure with Simes tests: every intersection containing a rejected hypothesis passes its local Simes test, so closed-testing control of the familywise error rate applies.

For $m\ge 1$ independent valid null p-values, the displayed inequality follows by applying the Benjamini-Hochberg procedure with all nulls true. Its false discovery rate then equals the probability of at least one rejection. Valid means $\mathrm{Pr}(p_i\le t)\le t$ for $0\le t\le 1$; independence is a sufficient hypothesis, and arbitrary dependence is not covered by this proof.

For the intersection of $m$ null hypotheses, reject when ${\min }_{i}m{p}_{(i)}/i\le \alpha$. The Simes inequality establishes a level-$\alpha$ test under independence of valid true-null p-values. The displayed quantity is its combined p-value.

Given valid p-values $q_J$ for constancy on intervals of length at least two, reject $I$ when ${\max }_{J\supseteq I}n{q}_J/|J|\le \alpha$. Every rejected true interval forces rejection of its maximal constant block. A union bound over the non-singleton maximal blocks gives familywise error rate at most $\alpha$, since their lengths sum to at most $n$. Singleton blocks have no tested interval in this formulation.

For a null hypothesis that is a union $H_0={\bigcup }_{i\in I}{H}_i$, an intersection-union test rejects $H_0$ only when every component $H_i$ is rejected. If each component test has level $\alpha$, then under any point of $H_0$ at least one component null is true, so the combined test also has level at most $\alpha$.

The Bonferroni correction tests each of $m$ hypotheses at level $\alpha /m$. The union bound then controls the familywise error rate by $\alpha$ under arbitrary dependence.

For positive deterministic weights $w_i$, the weighted Bonferroni correction rejects $H_i$ whenThe union bound over the true nulls controls the familywise error rate without an independence assumption.

Order $q_i=p_i/w_i$ increasingly. At rank $k$, reject the corresponding hypothesis whenand otherwise stop. The first ordered true null reduces FWER control to a weighted union bound; the shrinking denominator makes this procedure at least as powerful as the Weighted Bonferroni correction.

The Holm–Bonferroni method orders p-values and compares the $j$th smallest with $\alpha /(m-j+1)$, stopping at the first failed comparison. It controls the familywise error rate under arbitrary dependence.

If the Holm step-down procedure makes a false rejection, it must reject the first true null in the ordered list. All $m_0$ true nulls remain at that step, so its cutoff is at most $\alpha /m_0$. The union bound on their marginally valid p-values gives familywise error rate at most $\alpha$. This proves strong control under arbitrary dependence, and super-uniformity suffices.

Order $m$ p-values as $p_{(1)}\le \cdots \le p_{(m)}$ and take the largest $k$ with $p_{(k)}\le \alpha k/m$. The Benjamini-Hochberg procedure rejects the hypotheses corresponding to $p_{(1)},\dots ,p_{(k)}$ and controls the false discovery rate under independence and several forms of positive dependence.

Let $R$ be the number of rejections by the Benjamini-Hochberg procedure at level $\alpha$, and let $R_i$ be the rejection count after replacing $p_i$ by zero. Then $R_i\ge 1$ andIf $i$ was rejected, lowering its p-value does not change any ordered p-value beyond rank $R$, so $R_i=R$. Conversely, if $p_i\le \alpha R_i/m$, the other $R_i-1$ rejected values together with $p_i$ satisfy the original threshold; monotonicity gives $R=R_i$. For a valid true-null p-value independent of the others, conditioning on $R_i$ bounds the expectation of the right side by $\alpha /m$. Summing over true nulls proves false discovery rate at most $m_0\alpha /m$.

For the Benjamini-Hochberg procedure, replace a true-null p-value $P_i$ by zero and call the new rejection count $R^{(i)}$. The Benjamini-Hochberg leave-one-out identity identifies rejection with $R=r$ with the event $P_i\le \alpha r/m$ and $R^{(i)}=r$. If $P_i$ is uniform and independent of all the other p-values, each possible rejection count contributes $\alpha \mathbb{P}(R^{(i)}=r)/m$ to its expected false-discovery fraction. Summation gives $\alpha /m$ per true null and the displayed exact false discovery rate. Super-uniform independent nulls yield the corresponding upper bound.

The false discovery rate is $\mathbb{E}[V/(R\vee 1)]$, where $V$ is the number of rejected true null hypotheses and $R$ is the total number of rejections.

The familywise error rate is the probability of making at least one false rejection in a family of hypothesis tests.

For nonempty index sets forming a laminar family of sets, reject an intersection hypothesis when the displayed adjusted value is at most $\alpha$. The inclusion-maximal true sets are disjoint and have total size at most $m$. Any false rejection forces some maximal true $J$ to have $p_J\le \alpha |J|/m$. Validity and the union bound control the familywise error rate by $\alpha$, without independence or a complete closure.

Choose a level-$\alpha$ local test for every nonempty intersection $H_I={\bigcap }_{i\in I}{H}_i$. The closed testing procedure rejects an elementary hypothesis $H_i$ exactly when every intersection hypothesis containing it is rejected by its local test.

If any true elementary hypothesis is rejected by a closed testing procedure, then the intersection of all true null hypotheses must also be rejected. Its local test has level $\alpha$, so the familywise error rate is at most $\alpha$ under arbitrary dependence.

A joint hypothesis test imposes several parameter restrictions simultaneously and accounts for dependence among their estimators; separate one-parameter confidence intervals do not generally determine its result.

For a test with rejection region $R$, its power function isIts supremum over the null parameter space is the size of the test.

A level-$\alpha$ test is uniformly most powerful when its power is at least that of every other level-$\alpha$ test at every parameter value in the alternative.

If two tests have the same size and each has strictly greater power than the other at some alternative, neither test is uniformly most powerful.

For observed cell counts $O_j$ and null expected counts $E_j$, the statistic isWith $k$ specified null cell probabilities and sufficiently large expected counts, its null limit is ${\chi }_{k-1}^2$.

For an $r$ by $c$ contingency table, the statistichas asymptotically a chi-squared distribution with $(r-1)(c-1)$ degrees of freedom under independence.

For observed counts $y_{ij}$ and fitted counts ${\hat{\mu }}_{ij}=y_{i+}{y}_{+j}/y_{++}$ under independence, the likelihood-ratio statistic isUnder independence and standard large-sample regularity conditions, $G^2$ converges in distribution to ${\chi }_{(r-1)(c-1)}^2$.

Combining levels of a categorical variable changes both the null hypothesis and the degrees of freedom of an independence test. Association that is coherent across the combined levels may become easier to detect because the grouped table has fewer degrees of freedom, while associations that cancel within groups may disappear. Conclusions from the original and aggregated tables therefore need not agree.

For a two-way contingency table whose row and column variables are ordinal categorical variables, assign row scores $u_i$ and column scores $v_j$. The linear-by-linear association statistic measures the weighted covarianceAfter normalization, its square is asymptotically ${\chi }_1^2$ under independence. A signed, one-sided version tests a specified direction of ordinal association and can have greater statistical power than an omnibus independence test against a monotone alternative.

For an approximately normal estimator, the one-parameter Wald statisticis approximately standard normal under $H_0:\theta ={\theta }_0$. A two-sided test reports $2\Phi (-|Z|)$.

For an independent normal sample with known standard deviation $\sigma$, the signed Wald statistic for testing mean zero is $W=\sqrt{n}\bar{Y}/\sigma$. It has the standard normal distribution under the null hypothesis and distribution $N(\sqrt{n}\delta /\sigma ,1)$ under mean $\delta$. Its square has a chi-squared distribution; retaining the sign permits a one-sided test.

Independent normal arm means with variances $v_0,v_1,v_2$ give treatment-contrast Wald test statistics whose correlation coefficient is $v_0/\sqrt{(v_0+v_1)(v_0+v_2)}$. The common control mean creates positive dependence between comparisons.

For a regular $p$-parameter model, the Wald statistic for a candidate $\theta$ isUnder the true parameter it converges in distribution to ${\chi }_p^2$.

For a full-row-rank $k\times p$ matrix $R$ and the null hypothesis $R\theta =r$, putUnder the null, asymptotic normality and Slutsky theorem give $W_{n,R}\stackrel{d}{\to }{\chi }_k^2$.

In a regular scalar model under the null, Taylor expansion about the MLE givesA uniform law of large numbers for the observed information makes
$-{\ell }_n^{\prime \prime }(\tilde{\theta })/n$ and the Fisher information used in the Wald statistic converge to the same positive limit. Their ratio therefore converges in probability to one.

For $\overline{X}\sim N(\mu ,1/n)$ under $H_0:\mu =0$, the two-sided level-$\alpha$ test rejects when $|\sqrt{n}\overline{X}|>{\Phi }^{-1}(1-\alpha /2)$, and the observed p-value is

A likelihood-ratio test rejects where the alternative likelihood is sufficiently large relative to the null likelihood.

For a scalar parameter restricted to $\psi \ge 0$, test $\psi =0$ against $\psi >0$ with identifiable interior nuisance parameters. A locally quadratic regular likelihood has an efficient null score $Z\sim N(0,1)$, and the constrained fit projects the unconstrained local maximizer onto the nonnegative half-line. ConsequentlyFor a positive statistic, its asymptotic upper-tail p-value is half the ordinary one-degree chi-squared tail. Its 5% critical value is the 90th chi-squared percentile, approximately 2.7055. Additional boundary nuisance parameters or nonidentified mixture parameters can change this law.

Testing a single nonnegative random-effect variance against zero puts the null on the boundary of the parameter space. In the usual regular increasing-independent-groups limit, the likelihood-ratio test statistic has limit $\frac{1}{2}{\delta }_0+\frac{1}{2}{\chi }_1^2$, rather than ${\chi }_1^2$. A finite-sample simulation calibrated for the actual design avoids relying on this asymptotic approximation.

With fixed $X,Z$, compare $N(X\beta ,{\sigma }^{2}I)$ and $N(X\beta ,{\sigma }^{2}I+{\tau }^{2}Z{Z}^T)$ by maximizing the ordinary likelihood function in both models. Their likelihood-ratio test statistic is unchanged by $Y\mapsto Xb+cY$, $c>0$, because both maximized log-likelihoods change by the same $-n\log c$. Its null law can therefore be simulated using independent $N(0,I)$ responses. Comparing the observed statistic with simulated statistics by $(1+\#\{T_b\ge T_{obs}\})/(B+1)$ gives a conservative finite-sample Monte Carlo p-value, subject to correctly maximizing both likelihoods.

For two parameter values ${\theta }_1$ and ${\theta }_0$, the likelihood ratio is $f(x\mid {\theta }_1)/f(x\mid {\theta }_0)$.

For nested null and alternative parameter spaces, the likelihood-ratio test statistic isUnder the regularity conditions of Wilks theorem, its null limit is a chi-squared distribution whose degrees of freedom equal the difference in model dimensions.

A generalized likelihood-ratio test compares the likelihood maximized under the null with the likelihood maximized over the full parameter space.

Two equally numerous groups of independent counts with Poisson distributions have areas $a$ and $2a$. Under common density their group means are in ratio $1:2$. With group sums $S_1,S_2$ and $T=S_1+S_2$, the maximized alternative-to-null likelihood ratio isUse zero-count limits and $\Lambda =1$ when $T=0$. The regular Wilks theorem calibration has one degree of freedom. Exactly conditioning on $T$ gives $S_1\sim \mathrm{Binomial}(T,1/3)$ under the null; this supports small-count testing without the large-sample approximation.

The deviance of a fitted generalized linear model isFor regular nested models, the deviance reduction $D_0-D_1=2({\ell }_1-{\ell }_0)$ is asymptotically chi-squared with degrees of freedom equal to the difference in fitted dimensions.

The unscaled residual deviance of an exponential dispersion family compares its fitted mean with the saturated statistical model at a common dispersion parameter. Its normalization removes the factor $1/\varphi$ in the log-likelihood. In Poisson regression and binomial regression, $\varphi =1$, so it is twice the saturated-minus-fitted log-likelihood. Under suitable regularity, $D/\varphi$ has approximately a chi-squared distribution with the residual degrees of freedom; this is not an automatic accurate approximation for sparse responses. Nested deviance reductions with estimated dispersion lead to approximate F-tests after scaling by the dispersion estimate.

For independent unit-variance samples, testing equal means uses $Z=\sqrt{mn/(m+n)}(\bar{X}-\bar{Y})$ and rejects for large $|Z|$.

Likelihood-ratio tests of different null subspaces can have crossing rejection regions; comparing their quadratic-form geometry exhibits observations accepted by one and rejected by another.

The score test evaluates the likelihood gradient at the restricted estimator and scales its quadratic form by inverse Fisher information.

A restricted maximum-likelihood estimator maximizes likelihood over the null parameter space rather than the full model.

Under a regular simple $p$-parameter null, the normalized score converges to $N_p(0,I_1)$, so its information-standardized squared norm converges to ${\chi }_p^2$.

If $Z\sim N_p(0,I_p)$, then $Z^{T}Z={\sum }_{j=1}^p{Z}_j^2\sim {\chi }_p^2$.

For independent $N(\mu ,{\sigma }^2)$ observations, ${\sum }_i(X_i-\bar{X}{)}^2/{\sigma }^2$ has the ${\chi }_{n-1}^2$ distribution.

If $Q_m\sim {\chi }_m^2$, then $(Q_m-m)/\sqrt{2m}$ converges in distribution to $N(0,1)$.

For testing one simple hypothesis against another, a test that rejects for sufficiently large likelihood ratio has greatest power among all tests of no greater size.

For independent samples from an exponential distribution, a test of rate at most ${\lambda }_0$ against rate greater than ${\lambda }_0$ rejects for small sample sums. All simple-alternative likelihood ratios against ${\lambda }_0$ decrease with that sum; its gamma distribution calibrates the size at ${\lambda }_0$. Monotonicity of the rejection probability in the rate proves that this is a uniformly most powerful test for the composite hypotheses.

A one-parameter family has a monotone likelihood ratio in a statistic $T$ when $f(x\mid {\theta }_1)/f(x\mid {\theta }_0)$ is nondecreasing in $T(x)$ whenever ${\theta }_1>{\theta }_0$.

For a family with a monotone likelihood ratio in $T$, a level-$\alpha$ upper-tail test in $T$ is a uniformly most powerful test for the corresponding one-sided hypothesis.

For one observation from the logistic distribution, the size-$\alpha$ test of $H_0:\theta \le 0$ against $H_1:\theta >0$ rejects when