Past exam of the mathematics course of the University of Cambridge / 2015 / iii / Paper 36 / 1

Put $p=dP/d\mu$ and $q=dQ/d\mu$. Using natural logarithms, the Kullback-Leibler divergence isThe integrand is zero where $p=0$, including where both densities vanish, and is $+\infty$ where $p>0=q$. Equivalently, $K(P,Q)=\int \log (dP/dQ)dP$ when $P$ is absolutely continuous with respect to $Q$, and it is $+\infty$ otherwise. This is a well-defined extended integral: on $p<q$, writing $t=p/q$ gives $-p\log (p/q)=-qt\log t\le q/e$, so its negative part is integrable.

For Pinsker's inequality, an infinite Kullback-Leibler divergence makes the conclusion immediate. Assume it is finite. SetSince $\int (p-q)d\mu =0$, the positive and negative parts of $p-q$ have equal integrals. ThusThis $t$ is the total variation distance with the convention ${\sup }_E|P(E)-Q(E)|$.

We first prove the binary partition bound for relative entropy. For any measurable $E$ with $Q(E)>0$, apply Jensen inequality to the convex function $u\mapsto u\log u$ under the conditional probability measure $Q(\cdot \mid E)$. Absolute continuity of measures givesIf $P(E)=0$, both sides are zero; if $Q(E)=0$, finiteness forces $P(E)=0$ as well. Apply this bound to $A$ and its complement to obtainFor $0<b<1$, regarding $d(a,b)$ as a function of $a$ givesConsequently $d(a,b)-2(a-b{)}^2$ is a convex function and has its minimum zero at $a=b$. By continuity this also holds at $a=0,1$. For $b=0,1$, the finite case has $a=b$ and the remaining cases have infinite divergence. This proves the Bernoulli relative entropy lower bound in every case. Combining the bounds yields

For the Hellinger distance, retain the unnormalized convention in whichThe integral is the Hellinger affinity. In the finite-divergence case, $q/p>0$ for $P$-almost every point. Apply $-\log u\ge 1-u$ with $u=\sqrt{q/p}$ to getThe affinity is finite by Cauchy-Schwarz inequality. Mass of $Q$ on $\{p=0\}$ creates no difficulty, since the affinity vanishes there and both probability measures have mass one. The infinite-divergence case is again immediate. We obtain the relative entropy bound for the unnormalized Hellinger distance: