Randomization test

A randomization test compares a statistic with values obtained by applying transformations that preserve the joint distribution under the null hypothesis.

A conditional randomization test tests $Y\perp X\mid Z$ by resampling the tested variable from its conditional law given the remaining covariates. For independent sample rows, hold the observed $Y,Z$ fixed and draw each replacement vector $X^{(b)}$ from the true product conditional law of $X\mid Z$. Under the null hypothesis, the original $X$ and these replacements are conditionally exchangeable. Applying the same prespecified test statistic and fitting rule to every dataset therefore gives the valid rank-based Monte Carlo testThe add-one convention and conservative treatment of ties give finite-sample type I error control. Other resampling constructions are possible when they preserve the same conditional exchangeability.^[ref]

Replacing the true conditional law by an estimate does not automatically preserve that guarantee. For a counterexample without $Z$, let the true $X$ be a fair Bernoulli random variable and let $Y$ be constant, so independence holds. If the resampling model incorrectly uses $P(X=1)=0.99$ and the test statistic is $T=1-X$, the model assigns upper-tail probability $0.01$ whenever the observed $X=0$. That event actually occurs with probability $0.5$, so the limiting simulation test rejects far more often than level $0.05$ permits.

Estimated conditional models require a separate calibration or approximation argument. More resamples reduce Monte Carlo error but do not correct a wrong conditional law. The method tests conditional independence; interpreting it as a causal effect test requires the appropriate additional causal inference assumptions.

A permutation test uses relabelings of observations as its null-preserving transformations and computes a p-value from the resulting orbit of the test statistic.

A sign-flip randomization test independently multiplies observations by signs in $\{-1,1\}$. It is finite-sample exact for a zero-centered null when the joint distribution is invariant under those sign changes.