D (P ∥ Q) = D (P ∥ R) + D (R ∥ Q) + a \sum {P (a) - R (a)} lo g \frac{R ( a )}{Q ( a )} .

b

i

For

Q_{t} = (1 - t) Q^{*} + t Q^{0}

, convexity gives

Q_{t} \in E

. Minimality at

t = 0

implies

0 \leq \frac{d}{d t} D (P ∥ Q_{t})_{0 +} = \sum_{a} P (a) (1 - \frac{Q ^{0} ( a )}{Q ^{*} ( a )}) .

Since

P

has full support and the minimum is finite,

Q^{*}

has full support.

For

A^{'} = {a : P^{'} (a) > 0}

a \in A^{'} \sum P^{'} (a) (1 - \frac{P ( a ) Q ^{0} ( a )}{P ^{'} ( a ) Q ^{*} ( a )}) = 1 - a \in A^{'} \sum \frac{P ( a ) Q ^{0} ( a )}{Q ^{*} ( a )} \geq 1 - a \in A \sum \frac{P ( a ) Q ^{0} ( a )}{Q ^{*} ( a )} \geq 0,

using nonnegativity and part i.

Expanding the three divergences gives

D (P^{'} ∥ Q^{0}) + D (P^{'} ∥ P) - D (P^{'} ∥ Q^{*}) = \sum_{a \in A^{'}} P^{'} (a) lo g \frac{P ^{'} ( a ) Q ^{*} ( a )}{P ( a ) Q ^{0} ( a )} .

The inequality

lo g u \geq 1 - u^{- 1}

bounds this below by the nonnegative expression in part ii. Hence

D (P^{'} ∥ Q^{0}) + D (P^{'} ∥ P) \geq D (P^{'} ∥ Q^{*}) .

There are no discussions about this article yet.

There are currently no matching articles.