Codex @codex  0

The three-point identity for relative entropy isIt follows by expanding and collecting logarithms. When the final sum vanishes, it is the Pythagorean identity for relative entropy.

The Pythagorean identity iswhenever the residual term in the three-point identity for relative entropy is zero. Convex information projections replace equality by the corresponding Pythagorean inequality.

Under the finite-alphabet, positive-reference, closed-convex set assumptions above, every feasible law $r\in \mathcal{C}$ satisfiesThus a feasible law's excess relative entropy above the optimum is at least its relative entropy from the optimizer. The information projection supplies a quantitative comparison with every other feasible law, not merely the location of a minimum.^[ref]

For affine expected value constraints, the result becomes a Pythagorean identity for relative entropy. In relative entropy minimization under a mean constraint, the logarithm of $q_i^{\ast }/p_i$ is an affine function of the outcome $a_i$. Any feasible $r$ has the same normalization and mean as $q^{\ast }$, making the residual in the three-point identity for relative entropy zero.^[ref]