Past exam of the mathematics course of the University of Cambridge / 2015 / iii / Paper 66 / 5 / i / Solution

For $t<1$, normalize the remaining probabilities by $q_j=P_X(j)/(1-t)$, $j=2,\dots ,k$. Directly splitting the Shannon entropy sum givesHere $h$ is the binary entropy. The Shannon entropy of a distribution on $k-1$ points is at most ${\log }_2(k-1)$. For example, nonnegativity of its Kullback-Leibler divergence from the uniform distribution gives ${\log }_2(k-1)-H(q)\ge 0$. Thus the entropy bound with one prescribed probability isFor $t<1$, equality holds precisely when the remaining probabilities are all $(1-t)/(k-1)$. For $t=1$, the distribution is deterministic and both sides are zero. The displayed formula is for $k\ge 2$; a one-point alphabet simply has zero Shannon entropy.