 Solution

ID: past-exam-of-the-communication-theory-course-of-the-university-of-oxford/2010/2/b/ii/solution

Every positive atom has probability at most

Π

, so

- lo g_{2} p (x) \geq - lo g_{2} Π

. Averaging yields

H (X) = \sum_{x} p (x) (- lo g_{2} p (x)) \geq - lo g_{2} Π.

For

0 < Π \leq 1/2

, the supplied inequality

4Π \leq 4^{Π}

gives

2 + lo g_{2} Π \leq 2Π

. Rearranging proves

H (X) \geq - lo g_{2} Π \geq 2 (1 - Π) .

 New to topics? Read the docs here!