 Chernoff information for complementary Bernoulli distributions

Let $P$ be the Bernoulli distribution with success probability $0.2$ and $Q$ the Bernoulli distribution with success probability $0.8$. For $0\le s\le 1$,The two summands have product $0.16$. Their arithmetic mean is at least their geometric mean, with equality at $s=1/2$. ThusThe same argument gives $C=-\ln (2\sqrt{p(1-p)})$ for complementary parameters $0<p<1$. At $p=1/2$ the laws agree and $C=0$.

With equal prior probabilities, count the successes $K$ among $n$ observations. The likelihood ratio is $Q^{\otimes n}/P^{\otimes n}=4^{2K-n}$, so choose $Q$ when $K>n/2$ and $P$ when $K<n/2$. If $n$ is even, either decision or a fair tie randomization gives the same average error at $K=n/2$. Under $P$, $K$ has the binomial distribution with parameters $n,0.2$. Symmetry therefore gives the exact optimal average errorThe last term applies only for even $n$; it represents half the tie probability. Direct finite sums give $p_1^{\ast }=0.2$, $p_5^{\ast }=0.05792$ and $p_{20}^{\ast }\simeq 0.00157912$. Twenty observations reduce the optimal average error to about 0.158%. The finite-sample Chernoff error bound at $n=20$ is $\frac{1}{2}(0.8{)}^{20}\simeq 0.00576461$, a guarantee rather than the exact error.

Reducing the exact error from $p_{20}^{\ast }$ to $p_{50}^{\ast }\simeq 1.29363\times 10^{-6}$ illustrates the value of repeated evidence. The example assumes a single fixed coin law throughout the experiment. If repeated sensor readings share an unmodelled common error, increasing their number need not supply this much independent information.