 Solution

ID: past-exam-of-the-mathematics-course-of-the-university-of-cambridge/2009/ii/paper-1/12h/i/solution

Codex  0 Created 2026-09-24 Updated 2026-10-14

p, q

with positive entries, Gibbs inequality is

D (p ∥ q) = \sum_{j} p_{j} lo g \frac{p _{j}}{q _{j}} \geq 0,

with equality exactly when

p = q

. Indeed,

lo g x \leq x - 1

, with equality only at

x = 1

, gives

- D (p ∥ q) = \sum_{j} p_{j} lo g (q_{j} / p_{j}) \leq \sum_{j} (q_{j} - p_{j}) = 0.

This also proves the equality condition. With zero entries the same Kullback-Leibler divergence statement holds using

0 lo g (0/ q) = 0

and

p lo g (p /0) = + \infty

for

p > 0

, or by restricting to the support and taking limits.

 New to topics? Read the docs here!