Past exam of the mathematics course of the University of Cambridge / 2012 / iii / Paper 65 / 3 / i

Use base-two logarithms throughout, so information and entropy are in bits. The relative entropy of classical probability distributions isSet $0{\log }_2(0/q)=0$; if $p(x)>0$ and $q(x)=0$, define the value to be $+\infty$. In the finite case, $\ln u\le u-1$ givesHence $D(p\Vert q)\ge 0$, with equality exactly when $p=q$: equality in the logarithm inequality requires $q(x)=p(x)$ on the support, and equality in the last step leaves no mass outside it.