Solution

ID: past-exam-of-the-mathematics-course-of-the-university-of-cambridge/2008/ii/paper-1/4g/solution

If the possible values have probabilities , allowing zeros, the Shannon entropy in bits is , with . Each term is nonnegative, so , with equality precisely when one value has probability one.
For the upper bound, prove first that for : the function has derivative and its unique minimum, zero, at . Let be the number of positive probabilities. Applying this inequality to gives
The left side is , proving
Equality at the maximum requires and equality in every logarithmic inequality, so every . Conversely the uniform distribution attains it. For the maximum and minimum coincide at zero.

New to topics? Read the docs here!