= Chernoff information
{c}
{title2=$C(P,Q)$}
= Chernoff error exponent
{c}
{synonym}
<Chernoff information> measures how quickly repeated independent evidence can distinguish two fixed <probability distributions>. Suppose a <statistical hypothesis test> observes $n$ <independent and identically distributed random variables>, with common law $P$ under one <simple hypothesis> and $Q$ under the other. Give the two hypotheses fixed positive <prior probabilities>, and let $p_n^*$ be the smallest achievable average error probability. For distinct positive <probability mass functions> on a common <finite alphabet>,
$$
\lim_{n\to\infty}-\frac1n\ln p_n^*=C(P,Q),\qquad
C(P,Q)=-\min_{0\leq s\leq1}\ln\sum_xP(x)^sQ(x)^{1-s}.
$$
Here $\ln$ is the <natural logarithm>, so the <error exponent> is measured in <nats> per observation. This optimal rate is unchanged by choosing different fixed positive <prior probabilities>; the best decision rule and finite-sample error can still change.<Information Theory: From Coding to Learning, October 2022 draft by Yury Polyanskiy and others>[§16.1, Corollary 16.2 and equation (16.2), printed pp. 261–262: fixed-prior Bayesian error exponent]{ref}
The interpretation is $p_n^*=\exp(-nC(P,Q)+o(n))$: a larger <Chernoff information> means fewer observations are needed at the leading exponential scale. It does not assert the exact finite-sample identity $p_n^*=e^{-nC(P,Q)}$. The <finite-sample Chernoff error bound> and the explicit <Chernoff information for complementary Bernoulli distributions> make this distinction concrete. This is useful when designing repeated measurements or comparing sensors whose output <probability distributions> differ under two candidate conditions.
For <probability mass functions> with zero entries, define the affinity for $0<s<1$ and use $C(P,Q)=-\ln\inf_{0<s<1}\sum_xP(x)^sQ(x)^{1-s}$, taking endpoint limits when needed. This avoids an ambiguous $0^0$ convention. Identical laws give $C=0$; disjoint <probability supports> give $C=\infty$, because one observation separates them perfectly. The displayed finite-rate theorem concerns fixed laws with common positive support, not laws that change with $n$ or arbitrarily dependent repeated observations.
Back to article page
