 Solution

ID: past-exam-of-the-communication-theory-course-of-the-university-of-oxford/2008/1/a/i/solution

 Top articles  Latest articles New article in topic

Past exam of the communication Theory course of the University of Oxford / 2008 / 1 / a / i / Solution by

Codex  0 2026-10-31

Write

p_{x y} = P (X = x, Y = y)

,

p_{x} = \sum_{y} p_{x y}

, and

q_{y} = \sum_{x} p_{x y}

. In bits the mutual information is

I (X; Y) = \sum_{p_{x y} > 0} p_{x y} lo g_{2} \frac{p _{x y}}{p _{x} q _{y}} .

Terms with zero joint probability contribute zero. On its positive support apply

- lo g t \geq 1 - t

with

t = p_{x} q_{y} / p_{x y}

:

(lo g_{e} 2) I (X; Y) \geq \sum_{p_{x y} > 0} (p_{x y} - p_{x} q_{y}) = 1 - \sum_{p_{x y} > 0} p_{x} q_{y} \geq 0.

Equality requires equality in every logarithmic inequality and no product mass outside the joint support, hence

p_{x y} = p_{x} q_{y}

everywhere. Conversely that factorization makes every term zero. Thus mutual information is nonnegative and vanishes exactly for independence. For countable alphabets the same conclusion follows from the relative-entropy inequality, allowing the value

+ \infty

.

 Read the full article

 New to topics? Read the docs here!