= Chernoff information for complementary Bernoulli distributions
{c}
{title2=$C(\operatorname{Bern}(p),\operatorname{Bern}(1-p))=-\ln(2\sqrt{p(1-p)})$}
Let $P$ be the <Bernoulli distribution> with success probability $0.2$ and $Q$ the <Bernoulli distribution> with success probability $0.8$. For $0\leq s\leq1$,
$$
Z(s)=0.2^s0.8^{1-s}+0.8^s0.2^{1-s}.
$$
The two summands have product $0.16$. Their <arithmetic mean> is at least their <geometric mean>, with equality at $s=1/2$. Thus
$$
\boxed{C(P,Q)=-\ln0.8\simeq0.22314355\ \text{nats per observation}.}
$$
The same argument gives $C=-\ln(2\sqrt{p(1-p)})$ for complementary parameters $0<p<1$. At $p=1/2$ the laws agree and $C=0$.
\Image[/chernoff-complementary-coin-affinity.png]
{title=<Chernoff affinity> and <Chernoff information> for two complementary coin laws}
{id=image-chernoff-complementary-coin-affinity}
{description=Original computed example, with success probabilities $0.2$ and $0.8$. The affinity is minimized at $s=1/2$; the best negative-log affinity is $-\ln0.8$.}
With equal <prior probabilities>, count the successes $K$ among $n$ observations. The <likelihood ratio> is $Q^{\otimes n}/P^{\otimes n}=4^{2K-n}$, so choose $Q$ when $K>n/2$ and $P$ when $K<n/2$. If $n$ is even, either decision or a fair tie randomization gives the same average error at $K=n/2$. Under $P$, $K$ has the <binomial distribution> with parameters $n,0.2$. Symmetry therefore gives the exact optimal average error
$$
p_n^*=\sum_{k>n/2}\binom nk0.2^k0.8^{n-k}
+\frac12\mathbf1_{\{n\text{ even}\}}\binom n{n/2}0.2^{n/2}0.8^{n/2}.
$$
The last term applies only for even $n$; it represents half the tie probability. Direct finite <sums> give $p_1^*=0.2$, $p_5^*=0.05792$ and $p_{20}^*\simeq0.00157912$. \b[Twenty observations reduce the optimal average error to about 0.158%.] The <finite-sample Chernoff error bound> at $n=20$ is $\tfrac12(0.8)^{20}\simeq0.00576461$, a guarantee rather than the exact error.
\Image[/chernoff-coin-testing-error-decay.png]
{title=Exact repeated-coin error, the <finite-sample Chernoff error bound>, and the limiting <error exponent>}
{id=image-chernoff-coin-testing-error-decay}
{description=Original finite-binomial calculations under two fixed independent coin laws with equal priors. The first panel uses a <logarithmic scale>. The normalized negative log error approaches $C(P,Q)$; finite-sample error need not equal the simple exponential.}
Reducing the exact error from $p_{20}^*$ to $p_{50}^*\simeq1.29363\times10^{-6}$ illustrates the value of repeated evidence. The example assumes a single fixed coin law throughout the experiment. If repeated sensor readings share an unmodelled common error, increasing their number need not supply this much independent information.
Back to article page
