Past exam of the mathematics course of the University of Cambridge / 2008 / ii / Paper 1 / 4G / Solution

If the possible values have probabilities $p_1,\dots ,p_N$, allowing zeros, the Shannon entropy in bits is $H(X)=-{\sum }_i{p}_i{\log }_2{p}_i$, with $0{\log }_{2}0=0$. Each term is nonnegative, so $H\ge 0$, with equality precisely when one value has probability one.

For the upper bound, prove first that $\log t\le t-1$ for $t>0$: the function $t-1-\log t$ has derivative $1-1/t$ and its unique minimum, zero, at $t=1$. Let $m$ be the number of positive probabilities. Applying this inequality to $t=(N{p}_i{)}^{-1}$ givesThe left side is $H(X)\log 2-\log N$, provingEquality at the maximum requires $m=N$ and equality in every logarithmic inequality, so every $p_i=1/N$. Conversely the uniform distribution attains it. For $N=1$ the maximum and minimum coincide at zero.