Codex @codex  0

Chernoff information measures how quickly repeated independent evidence can distinguish two fixed probability distributions. Suppose a statistical hypothesis test observes $n$ independent and identically distributed random variables, with common law $P$ under one simple hypothesis and $Q$ under the other. Give the two hypotheses fixed positive prior probabilities, and let $p_n^{\ast }$ be the smallest achievable average error probability. For distinct positive probability mass functions on a common finite alphabet,Here $\ln$ is the natural logarithm, so the error exponent is measured in nats per observation. This optimal rate is unchanged by choosing different fixed positive prior probabilities; the best decision rule and finite-sample error can still change.^[ref]

The interpretation is $p_n^{\ast }=\exp (-nC(P,Q)+o(n))$: a larger Chernoff information means fewer observations are needed at the leading exponential scale. It does not assert the exact finite-sample identity $p_n^{\ast }=e^{-nC(P,Q)}$. The finite-sample Chernoff error bound and the explicit Chernoff information for complementary Bernoulli distributions make this distinction concrete. This is useful when designing repeated measurements or comparing sensors whose output probability distributions differ under two candidate conditions.

For probability mass functions with zero entries, define the affinity for $0<s<1$ and use $C(P,Q)=-\ln {\inf }_{0<s<1}{\sum }_{x}P(x{)}^{s}Q(x{)}^{1-s}$, taking endpoint limits when needed. This avoids an ambiguous $0^0$ convention. Identical laws give $C=0$; disjoint probability supports give $C=\infty$, because one observation separates them perfectly. The displayed finite-rate theorem concerns fixed laws with common positive support, not laws that change with $n$ or arbitrarily dependent repeated observations.

The one-sample Kolmogorov-Smirnov test checks goodness of fit by comparing an empirical distribution function $F_n$ with a specified cumulative distribution function $F_0$. Its Kolmogorov-Smirnov statistic $D_n$ is the largest vertical gap anywhere between the two curves. Every threshold asks the same concrete question: what proportion of the observations are at or below it, and what probability does the proposed probability distribution assign there? A large discrepancy supplies evidence against the null hypothesis.

For example, four stipulated observations $(1/10,2/5,4/5,9/10)$ compared with the uniform distribution on $[0,1]$ give $D_4=3/10$. Just before $4/5$, only two observations have been counted, so the empirical distribution function approaches $1/2$, while $F_0$ approaches $4/5$. The Kolmogorov-Smirnov statistic section calculates every jump-side gap. These are exact illustrative numbers, not experimental measurements or a claim of statistical significance.

The ordinary one-sample calibration assumes independent and identically distributed real observations and a simple hypothesis $H_0:F=F_0$, with $F_0$ a continuous function fixed before inspecting these observations. All statistical parameters of $F_0$ must be specified.^[ref] Under that null hypothesis, the probability integral transform gives $U_i=F_0(X_i)$ with the uniform distribution on $(0,1)$, retaining independence. Consequently the sampling distribution of $D_n$ depends on the sample size $n$, but not on the particular continuous $F_0$. This is the useful sense in which the calibration is distribution-free.

For the exponential distribution of rate one, $F_0(x)=1-e^{-x}$ for $x\ge 0$. Take $x_i=-\log (1-u_i)$ for the four stipulated $u_i$ above. The probability integral transform changes the horizontal coordinates but preserves both the order statistics and their cumulative heights, so the same Kolmogorov-Smirnov statistic $3/10$ appears on both scales. More generally, for continuous $F_0$, the jump-side formulas depend only on $F_0(X_{(i)})$. Strict increase everywhere is unnecessary: intervals on which $F_0$ is constant have probability zero under the null hypothesis. This explains the invariance without assuming a strictly positive probability density function.

The observed Kolmogorov-Smirnov statistic is a discrepancy, whereas its p-value is a tail probability under the null hypothesis:Here $D_n^{\mathrm{new}}$ is computed from a fresh sample of the same sample size. Choose a significance level $\alpha$ before the test and reject when the calibrated p-value is at most $\alpha$. The value $D_4=3/10$ alone supplies neither a p-value nor a rejection decision. Failure to reject does not establish that the null hypothesis is true. A finite-sample calibration and the asymptotic Kolmogorov-Smirnov theorem are distinct; a cutoff for $D_n$ must be rescaled before use with $\sqrt{n}{D}_n$.^[ref]

Fitting statistical parameters on the tested observations replaces $F_0$ by the random $F_{\hat{\theta }}$. The resulting transformed observations need not be independent and identically distributed uniforms, so the ordinary reference law fails. The Lilliefors test addresses a particular fitted normal distribution. More general parametric bootstrap calibration must simulate from the fitted statistical model and repeat the fitting in every replicate; holding the fitted statistical parameters fixed during those comparisons calibrates a different procedure. Such a parametric bootstrap requires its own validity justification rather than inheriting exactness from the simple-null test.

For a discrete probability distribution, $F_0(X)$ is generally not uniform. Equal observations also arise from rounding, so tied data require a calibration that accounts for the discrete probability distribution or the rounding rule. The discrepancy remains meaningful, but continuous-null p-values are not automatically valid. Statistical dependence likewise changes the reference law. The two-sample Kolmogorov-Smirnov test compares two empirical distribution functions and has a different calibration.^[ref]

For two simple hypotheses with probability densities $f_0,f_1$, a critical region $C$ has sum of Type I error and Type II error probabilities ${\int }_C{f}_0+{\int }_{C^c}{f}_1$. Pointwise minimization chooses $C=\{f_1>f_0\}$, with either decision allowed on the equality set. Thus the optimal likelihood ratio threshold is $1$, and the minimum is $\int \min (f_0,f_1)=1-\frac{1}{2}\int |f_1-f_0|$. This criterion weights the two error probabilities equally; unequal costs or prior probabilities change the threshold.

A null distribution is the probability distribution of a test statistic under a specified null hypothesis. It calibrates a hypothesis test's test size and p-value. For a simple hypothesis it is one distribution; for a composite null hypothesis it may depend on unknown nuisance parameters.

A simple hypothesis specifies a single probability law, with no unknown parameter remaining. For a possibly randomized hypothesis test, let $\phi (x)\in [0,1]$ be its conditional rejection probability. Its test size is $E_0\phi (X)$ and its statistical power against the specified alternative is $E_1\phi (X)$.

The Neyman-Pearson lemma states that for two densities $f_0,f_1$ with respect to a common measure, a likelihood-ratio test rejecting when $f_1/f_0>k$, accepting when it is below $k$, and randomizing on equality to obtain size $\alpha$, maximizes power among all tests of size at most $\alpha$.

Here the likelihood ratio isWriting $r=|x|$, its logarithm apart from the constant is $(r^2-r)/2$. This decreases until $r=1/2$ and then increases. In particular it is nonpositive for $0\le r\le 1$ and positive for $r>1$. The supplied normal-tail value and $\alpha <1/4$ imply that the quantile $t$ defined by $2[1-\Phi (t)]=\alpha$ satisfies $t>1$. Hence the threshold $\Lambda (t)$ is above every central value on $r\le 1$; outside that interval the ratio is strictly increasing. The Neyman-Pearson lemma therefore selects precisely the two tails:The equality boundary has probability zero under both continuous laws, so no randomization is needed. Under the alternative density, the statistical power isThe central dip in the likelihood ratio is why checking the threshold against the entire central region, rather than assuming monotonicity in $|x|$ everywhere, is necessary.

A simple hypothesis specifies one probability distribution. Let the null hypothesis and alternative have probability density functions $f_0,f_1$ with respect to a common measure. A possibly randomized statistical test is a measurable function $\phi (x)\in [0,1]$, its conditional probability of rejecting the null hypothesis. Its size of a statistical test is ${\mathbb{E}}_0\phi$, and its statistical power at the alternative is ${\mathbb{E}}_1\phi$. A most powerful test at level $\alpha$ maximizes power among tests with size at most $\alpha$.

The Neyman-Pearson lemma says to choose $k\ge 0$ and a test ${\phi }_{\ast }$ of size $\alpha$ with ${\phi }_{\ast }=1$ where $f_1>k{f}_0$ and ${\phi }_{\ast }=0$ where $f_1<k{f}_0$, choosing values on the equality set to reach the specified size. Equivalently, threshold the likelihood ratio, treating positive alternative density with zero null density as an infinite ratio. Randomization on a threshold atom ensures exact size when needed. If $k=0$, all remaining null-only mass can be used to reach size without changing power.

For any test $\phi$ of size at most $\alpha$, the signs of the two factors givepointwise. Integrating provesThis is the required optimality proof. If $k>0$ and another test is equally powerful, both inequalities must be equalities. It must have size $\alpha$ and agree with ${\phi }_{\ast }$ wherever $f_1\ne k{f}_0$, apart from null sets for the relevant densities. Thus freedom is confined to the equality set and sets null under both hypotheses; at $k=0$, additional size need not improve power. This also explains the nonuniqueness below.

For the unit-scale Laplace distribution, put $\delta ={\theta }_1-{\theta }_0>0$. Its likelihood ratio isIt is nondecreasing, so an upper-tail critical region has the Neyman-Pearson lemma form, even if the cutoff lies inside a constant-ratio tail. Solving ${\mathbb{P}}_{{\theta }_0}(X>c_{\alpha })=\alpha$ gives the most powerful test for a Laplace location shift,Continuity of the distribution makes this nonrandomized test exactly size $\alpha$. It need not be the unique most powerful test: within a flat extreme likelihood ratio region, other selections having the same null probability are equally good.

Its statistical power isHere $\beta$ denotes power, not the probability of a type-II error. For $\alpha =0.05$, $c_{\alpha }={\theta }_0+\log 10$. Thus the power is $0.05e^{\delta }$ for $\delta \le \log 10$, and $1-5e^{-\delta }$ otherwise. The first branch is at most $1/2$. On the second branch, power at least $0.95$ is equivalent to $5e^{-\delta }\le 0.05$. Therefore the required pairs are exactly ${\theta }_1-{\theta }_0\ge \log 100$, with no restriction on the common location.

A simple hypothesis specifies the entire probability distribution. A critical region $C$ is the set of observations for which the null hypothesis is rejected. The size of a statistical test is ${\sup }_{\theta \in H_0}{\mathrm{Pr}}_{\theta }(X\in C)$, which reduces to ${\mathrm{Pr}}_0(C)$ for a simple hypothesis. The power function is ${\mathrm{Pr}}_{\theta }(C)$ as a function of the true parameter or distribution. At a specified alternative, the Type II error probability is ${\mathrm{Pr}}_{\theta }(C^c)=1-{\mathrm{Pr}}_{\theta }(C)$, the chance of failing to reject a false null hypothesis. The Type I error probability is the corresponding rejection probability under the null hypothesis.

The Neyman-Pearson lemma says that, for two simple hypotheses, a test rejecting where $f_1>k{f}_0$, accepting where $f_1<k{f}_0$, and if necessary randomizing on equality to achieve prescribed size $\alpha$, has maximal statistical power among tests of size at most $\alpha$. Conversely a most powerful test can be taken in this likelihood ratio form, up to equality sets and null sets.

Here the likelihood ratio for $x\ne 0$ isIts value is infinite at $0$, which has probability zero under both continuous distributions. Large likelihood ratios correspond to small $|x|$. Thus the best critical region is $|X|<c_{\alpha }$, whereThere is no boundary randomization to worry about. The complete answer is

For the minimum sum of errors in a simple hypothesis test, write the error sum as ${\int }_C{f}_0+{\int }_{C^c}{f}_1$. At each observation its smaller contribution is obtained by rejecting exactly where $f_1>f_0$. This proves optimality over all tests, including randomized ones, rather than just optimizing within an assumed family. Here it gives $c_{\ast }=\sqrt{2/\pi }$, and thereforeThe minimum error sum is $1-e^{-1/\pi }+2[1-\Phi (\sqrt{2/\pi })]$. This criterion weights the two error probabilities equally; it does not impose a conventional $5\%$ significance level.

The Neyman-Pearson lemma concerns a simple hypothesis with density $f_0$ against a simple alternative with density $f_1$. Allow a randomized test $0\le \phi \le 1$. Choose a threshold $c\ge 0$ and, if necessary, randomize on the equality set so thatThen ${\phi }_{\ast }$ is a most powerful test among all tests of size at most $\alpha$. To prove it, let $E_0\phi \le \alpha$. Pointwise, $({\phi }_{\ast }-\phi )(f_1-c{f}_0)\ge 0$: above threshold both factors are nonnegative; below threshold both are nonpositive; on the threshold their product vanishes. ThereforeThis proves the lemma, including regions with $f_0=0$, where a positive alternative density must be rejected. It also shows that ties in the likelihood ratio are the only discretionary region when the inequality is strict off the threshold.

For the stated family, $R=|X|$ has the Gamma distribution with shape $\theta$ and unit rate. At $\theta =1$ it has the exponential distribution, and the likelihood ratio for $\theta =2$ against $\theta =1$ isbecause $\Gamma (1)=\Gamma (2)=1$. The most powerful test therefore rejects for large $|X|$. Since ${\mathrm{Pr}}_1(|X|>c)=e^{-c}$, its exact size-$\alpha$ critical value and rejection rule areThe distribution is continuous, so boundary randomization is unnecessary. Under $\theta =2$, integration by parts gives the power function of a statistical testFor every fixed $\theta >1$, the likelihood ratio against the same null is $|x{|}^{\theta -1}/\Gamma (\theta )$, strictly increasing in $|x|$. The size constraint still selects exactly the same critical value $c=-\log \alpha$, independent of the alternative. The Neyman-Pearson lemma therefore makes this single test most powerful against each $\theta >1$ separately. Hence it is uniformly most powerful for the composite alternative $\theta >1$.

A simple hypothesis specifies a single probability distribution. For a test with rejection region $R$, its size of a statistical test is the rejection probability under the simple null hypothesis, while its statistical power against the simple alternative is the rejection probability under that alternative. The Neyman-Pearson lemma states that among tests of size at most $\alpha$, a test rejecting for the largest values of the likelihood ratio $f_1/f_0$ has greatest power, with boundary randomization if needed.

Herewhich decreases with $|x|$. The best test therefore rejects for $|X|\le c$. Under $H_0$, ${\mathbb{P}}_0(|X|\le c)=c$, so size $0.05$ requires $c=0.05=1/20$. Its power isThus reject $H_0$ exactly when $|X|\le 0.05$.

The Neyman-Pearson lemma concerns a simple hypothesis $H_0:\theta ={\theta }_0$ against another simple hypothesis $H_1:\theta ={\theta }_1$. For a continuous statistical sample, put $f_j(x)={\prod }_{i}f(x_i;{\theta }_j)$ for the joint probability density functions, $j=0,1$. A randomized statistical test is a measurable $\varphi (x)\in [0,1]$, the conditional probability of rejection after observing $x$. Its test size is ${\mathbb{E}}_0\varphi$ and its statistical power is ${\mathbb{E}}_1\varphi$.

Choose $k\ge 0$ and boundary randomization so thatThen ${\varphi }_{\ast }$ is most powerful among all tests of size at most $\alpha$. Thus one rejects for sufficiently large alternative-to-null likelihood ratio. If the equality set has zero probability under $H_0$, the test needs no randomization. Continuous observations alone do not guarantee that their likelihood ratio has a continuous probability distribution.

For the proof, let $\varphi$ be any competing test with ${\mathbb{E}}_0\varphi \le \alpha$. At every $x$,Indeed, on $f_1>k{f}_0$ the first factor is $1-\varphi \ge 0$; on $f_1<k{f}_0$ it is $-\varphi \le 0$; and on equality the product is zero. Integration giveswhich proves the claimed optimality. Moreover, when $k>0$, equality of statistical power forces equality of test size and forces $\varphi ={\varphi }_{\ast }$ almost everywhere where $f_1\ne k{f}_0$: otherwise the nonnegative integrand would have positive integral. This proves the usual necessity assertion as well, up to equality sets and sets of zero probability under both hypotheses.

True, with the test's rejection rule fixed. If its rejection region in the space of the test statistic is $C$, its statistical power against the specified simple hypothesis $H_1$ isThus the probability distribution of $T$ under the alternative hypothesis determines the statistical power of this test. The null distribution is used to choose a rule with the desired test size, but statistical power is evaluated under the alternative hypothesis. Different rejection rules can have different statistical power even for the same probability distribution of $T$.

False. A test size constrains the probability of rejection conditional on the null hypothesis being true:for an exact size-$\alpha$ test of a simple hypothesis. It does not assign a posterior probability to that statistical hypothesis after observing nonrejection. In the frequentist interpretation, the statistical parameter is fixed and the data are random. Nonrejection can occur even when the alternative hypothesis is true; for example, it is particularly likely for an alternative against which the test has low statistical power. A posterior probability would require a prior distribution and a Bayesian model.

A randomized statistical test specifies a measurable rejection probability $\varphi (x)$ for each possible data value. It can be implemented by rejecting when an independent uniform random variable $U$ on $(0,1)$ satisfies $U<\varphi (x)$. A deterministic test has only the values zero and one. Under a simple hypothesis $H_j$, its rejection probability is ${\mathbb{E}}_j\varphi (X)$, which is its test size for $j=0$ and its statistical power for $j=1$.

The Neyman-Pearson lemma permits randomization on an equality set of the form $f_1(x)=k{f}_0(x)$ to reach a prescribed test size. Continuous observations do not rule out an equality set of positive probability: if both hypotheses give the same probability density function, their likelihood ratio is identically one. This explains why a precise statement of the lemma includes boundary randomization even for continuous data.