Past exam of the communication Theory course of the University of Oxford / 2007 / 1 / a / Solution

Use the convention $0{\log }_{2}0=0$. On the positive support of $p$, the given logarithm inequality givesThe logarithm is strictly concave, so its supporting line at one has equality only there. More explicitly, the given global bound forces $c=1/{\log }_{e}2$ by differentiation at one, and $x-1-{\log }_{e}x$ vanishes only at one. Equality above therefore requires $p_r=q_r$ on the positive support, and no mass of $q$ outside it. Thus equality holds exactly when $p=q$.

For the uniform distribution $q_r=1/n$, the divergence is $D(p,q)={\log }_{2}n-H(p)$. Its nonnegativity gives $\boxed{H(p)\le {\log }_{2}n}$, with equality precisely for the uniform law.