Choose a most probable symbol

x_{*}

, and let

B = 1_{{X = x_{*}}}

H (B) = h_{2} (Π) = - Π lo g_{2} Π - (1 - Π) lo g_{2} (1 - Π) .

Since

B

X

, part (a)(ii) gives

H (X) \geq h_{2} (Π)

. On

[1/2, 1]

h_{2}

lies above the chord joining

(1/2, 1)

and

(1, 0)

. That chord has height

2 (1 - Π)

. Therefore

H (X) \geq h_{2} (Π) \geq 2 (1 - Π) .

The endpoint

Π = 1

is included by continuity and corresponds to a deterministic source.

ii

Solution

Every positive atom has probability at most

Π

, so

- lo g_{2} p (x) \geq - lo g_{2} Π

. Averaging yields

H (X) = \sum_{x} p (x) (- lo g_{2} p (x)) \geq - lo g_{2} Π.

For

0 < Π \leq 1/2

, the supplied inequality

4Π \leq 4^{Π}

gives

2 + lo g_{2} Π \leq 2Π

. Rearranging proves

H (X) \geq - lo g_{2} Π \geq 2 (1 - Π) .

There are no discussions about this article yet.

There are currently no matching articles.