 Three-point identity for relative entropy

ID: three-point-identity-for-relative-entropy

P, Q, R

with compatible supports,

D (P ∥ Q) = D (P ∥ R) + D (R ∥ Q) + \sum_{a} {P (a) - R (a)} lo g \frac{R ( a )}{Q ( a )} .

When the final term vanishes, this is the Pythagorean identity for relative entropy.

 New to topics? Read the docs here!