 Binary relative entropy

ID: binary-relative-entropy

The binary relative entropy is the Kullback-Leibler divergence between Bernoulli distributions:

kl (a, b) = a lo g \frac{a}{b} + (1 - a) lo g \frac{1 - a}{1 - b} .

It is the exact exponent obtained by optimizing the Chernoff bound for a Bernoulli mean.

 New to topics? Read the docs here!