= Information projection
{title2=$q^*=\arg\min_{q\in\mathcal C}D(q\Vert p)$}
{wiki}
= I-projection
{c}
{synonym}
An <information projection> chooses the <probability distribution> that satisfies a constraint while departing least from a reference law in <relative entropy>. The constraint might specify an <expected value>, several marginal <probability distributions>, or another <convex set> of permissible laws. The reference is held fixed: the variable is the first argument of $D(q\Vert p)$. Reversing the arguments gives a different minimization problem.
For a finite <alphabet> $\mathcal A$, take a positive reference <probability mass function> $p(a)>0$ and a nonempty closed <convex set> $\mathcal C$ in the <probability simplex>. Then
$$
\boxed{q^*=\arg\min_{q\in\mathcal C}D(q\Vert p),\qquad
D(q\Vert p)=\sum_{a:q(a)>0}q(a)\ln\frac{q(a)}{p(a)}.}
$$
The minimum exists and its minimizing <probability distribution> is unique. In this article, <natural logarithms> give <nats>. Using base-two <logarithms> rescales the objective without changing the minimizer.<Information Theory and Statistics: A Tutorial by Imre Csiszár and Paul C. Shields>[§3, printed pp. 440–441: finite-alphabet information projection onto a closed convex set]{ref}
If $p$ already meets the constraint, its <information projection> is $p$ itself by <relative entropy nonnegativity>. If not, a constraint can change the relative weights of outcomes without specifying every weight individually. <Relative entropy minimization under a mean constraint> gives an especially useful example: it tilts the reference law rather than choosing an arbitrary distribution with the right average.
Back to article page
