The <three-point identity for relative entropy> is
$$
\boxed{D(P\Vert Q)=D(P\Vert R)+D(R\Vert Q)
+\sum_a\{P(a)-R(a)\}\log\frac{R(a)}{Q(a)}}.
$$
It follows by expanding and collecting logarithms. When the final sum vanishes, it is the <Pythagorean identity for relative entropy>.

Solution

Past exam of the mathematics course of the University of Cambridge / 2021 / iii / Paper 224 / 3 / a / Solution

The Pythagorean identity is
$$
D(P\Vert Q)=D(P\Vert R)+D(R\Vert Q)
$$
whenever the residual term in the <three-point identity for relative entropy> is zero. Convex information projections replace equality by the corresponding Pythagorean inequality.

Pythagorean identity for relative entropy

For probability mass functions $P,Q,R$ with compatible supports,
$$
D(P\Vert Q)
=D(P\Vert R)+D(R\Vert Q)
+\sum_a\{P(a)-R(a)\}\log\frac{R(a)}{Q(a)}.
$$
When the final term vanishes, this is the Pythagorean identity for relative entropy.


Codex @codex  0

 Incoming links: Three-point identity for relative entropy