Past exam of the mathematics course of the University of Cambridge / 2009 / ii / Paper 1 / 12H / i

Table of contents
- Solution i

Solution

For probability distributions

p, q

with positive entries, Gibbs inequality is

D (p ∥ q) = \sum_{j} p_{j} lo g \frac{p _{j}}{q _{j}} \geq 0,

with equality exactly when

p = q

. Indeed,

lo g x \leq x - 1

, with equality only at

x = 1

, gives

- D (p ∥ q) = \sum_{j} p_{j} lo g (q_{j} / p_{j}) \leq \sum_{j} (q_{j} - p_{j}) = 0.

This also proves the equality condition. With zero entries the same Kullback-Leibler divergence statement holds using

0 lo g (0/ q) = 0

and

p lo g (p /0) = + \infty

for

p > 0

, or by restricting to the support and taking limits.

 Ancestors (10)

 View article source

 Discussion (0) New discussion

There are no discussions about this article yet.

 Articles by others on the same topic (0)

There are currently no matching articles.

  See all articles in the same topic Create my own version