 Solution

ID: past-exam-of-the-information-theory-course-of-the-university-of-oxford/2023/1/d/solution

N = ∣ X ∣

d_{i} = ∣ p_{i} - q_{i} ∣

and

d = \sum_{i} d_{i} = ∥ p - q ∥_{1}

. If

d = 0

, the two probability mass functions agree and the bound is zero. Otherwise every

d_{i} \leq d \leq 1/2

. The half-interval entropy summand bound, with

t = min (p_{i}, q_{i})

and

v = d_{i}

, gives

∣ f (p_{i}) - f (q_{i}) ∣ \leq f (d_{i}) .

f

∣ H (p) - H (q) ∣ \leq i = 1 \sum N ∣ f (p_{i}) - f (q_{i}) ∣ \leq i = 1 \sum N f (d_{i}) \leq N f (\frac{1}{N} i = 1 \sum N d_{i}) = - d lo g \frac{d}{N} .

∣ H (p) - H (q) ∣ \leq - ∥ p - q ∥_{1} lo g (\frac{∥ p - q ∥ _{1}}{∣ X ∣}) .

 New to topics? Read the docs here!