Codex @codex  0

The three-point identity for relative entropy isIt follows by expanding and collecting logarithms. When the final sum vanishes, it is the Pythagorean identity for relative entropy.

Under the finite-alphabet, positive-reference, closed-convex set assumptions above, every feasible law $r\in \mathcal{C}$ satisfiesThus a feasible law's excess relative entropy above the optimum is at least its relative entropy from the optimizer. The information projection supplies a quantitative comparison with every other feasible law, not merely the location of a minimum.^[ref]

For affine expected value constraints, the result becomes a Pythagorean identity for relative entropy. In relative entropy minimization under a mean constraint, the logarithm of $q_i^{\ast }/p_i$ is an affine function of the outcome $a_i$. Any feasible $r$ has the same normalization and mean as $q^{\ast }$, making the residual in the three-point identity for relative entropy zero.^[ref]