= Pythagorean inequality for information projection
{c}
{title2=$D(r\Vert p)\geq D(r\Vert q^*)+D(q^*\Vert p)$}
Under the finite-<alphabet>, positive-reference, closed-<convex set> assumptions above, every feasible law $r\in\mathcal C$ satisfies
$$
\boxed{D(r\Vert p)\geq D(r\Vert q^*)+D(q^*\Vert p).}
$$
Thus a feasible law's excess <relative entropy> above the optimum is at least its <relative entropy> from the optimizer. The <information projection> supplies a quantitative comparison with every other feasible law, not merely the location of a minimum.<6.441 Information Theory (Spring 2016 lecture notes) by Yury Polyanskiy and others/Chapter 12: Information Projection and Large Deviation>[§12.2, Theorem 12.2, printed p. 133]{ref}
For affine <expected value> constraints, the result becomes a <Pythagorean identity for relative entropy>. In <relative entropy minimization under a mean constraint>, the <logarithm> of $q_i^*/p_i$ is an affine function of the outcome $a_i$. Any feasible $r$ has the same normalization and <mean> as $q^*$, making the residual in the <three-point identity for relative entropy> zero.<Information Theory and Statistics: A Tutorial by Imre Csiszár and Paul C. Shields>[§3, Theorem 3.2 and equation (3.2), printed pp. 442–443]{ref}
Back to article page
