Information projection ($q^{\ast }=\arg {\min }_{q\in \mathcal{C}}D(q\Vert p)$)

An information projection chooses the probability distribution that satisfies a constraint while departing least from a reference law in relative entropy. The constraint might specify an expected value, several marginal probability distributions, or another convex set of permissible laws. The reference is held fixed: the variable is the first argument of $D(q\Vert p)$. Reversing the arguments gives a different minimization problem.

For a finite alphabet $\mathcal{A}$, take a positive reference probability mass function $p(a)>0$ and a nonempty closed convex set $\mathcal{C}$ in the probability simplex. ThenThe minimum exists and its minimizing probability distribution is unique. In this article, natural logarithms give nats. Using base-two logarithms rescales the objective without changing the minimizer.^[ref]

If $p$ already meets the constraint, its information projection is $p$ itself by relative entropy nonnegativity. If not, a constraint can change the relative weights of outcomes without specifying every weight individually. Relative entropy minimization under a mean constraint gives an especially useful example: it tilts the reference law rather than choosing an arbitrary distribution with the right average.

Let the possible numerical outcomes be $-1,0,1$, and give the reference law $p$ uniform distribution on a finite set: $p=(1/3,1/3,1/3)$. We want a law $q$ with expected value $1/2$. Write the three probabilities in outcome order as $(q_{-},q_0,q_{+})$. The mean constraint is $q_{+}-q_{-}=1/2$, while normalization requires $q_{-}+q_0+q_{+}=1$.

These conditions leave a whole line of possible probability distributions:For example, $(0,1/2,1/2)$ and $(1/4,0,3/4)$ both have the required mean. The constraint alone does not choose between them. Information projection selects the point on this line with the smallest relative entropy from the reference.

Exponential tilting gives weights proportional to $e^{-\theta },1,e^{\theta }$. Set $t=e^{\theta }>0$ and multiply these weights by $t$ to obtain $q^{\ast }=(1,t,t^2)/(1+t+t^2)$. Enforcing the mean givesThe positive root is $t=(1+\sqrt{13})/2$, henceThe most favored outcome becomes $1$, but all three remain possible. This is an exact constructed example with a stipulated reference law and constraint, not a fitted distribution from measurements.

The word "projection" does not mean ordinary perpendicular projection in the plotted Euclidean space. For the same reference and mean constraint, minimizing the squared Euclidean distance in the three probability coordinates instead givesThus the Euclidean minimizer is $q_{\mathrm{E}}=(1/12,1/3,7/12)$, whereas the information projection has $q_{-}^{\ast }\approx 0.116204$. The squared Euclidean objective at its own minimum is $1/8$; its relative entropy is approximately $0.210918$ nats, larger than the optimum $0.197378$ nats. Both distributions satisfy the same constraint.

The reference and objective encode what "least change" means. Relative entropy is asymmetric and is not a metric; its minimization answers a different question from linear least squares. The useful geometric analogy comes from a Pythagorean inequality for information projection, rather than from an assertion that the colored contours are circles.

For a uniform distribution on a finite set of $m$ outcomes, direct expansion of relative entropy giveswhere $H_e$ is information entropy in nats. Therefore minimizing relative entropy from this particular reference is equivalent to maximizing information entropy subject to the same constraint. This statement depends on the uniform reference; an arbitrary reference contributes its own weights.

In the three-outcome example, $H_e(q^{\ast })=\ln 3-D(q^{\ast }\Vert p)\approx 0.901235$ nats. The mean fixes one aggregate quantity, and information projection chooses how to distribute the remaining uncertainty. This extends maximum entropy on a finite alphabet, whose unconstrained maximizer is the uniform law itself.

Under the finite-alphabet, positive-reference, closed-convex set assumptions above, every feasible law $r\in \mathcal{C}$ satisfiesThus a feasible law's excess relative entropy above the optimum is at least its relative entropy from the optimizer. The information projection supplies a quantitative comparison with every other feasible law, not merely the location of a minimum.^[ref]

For affine expected value constraints, the result becomes a Pythagorean identity for relative entropy. In relative entropy minimization under a mean constraint, the logarithm of $q_i^{\ast }/p_i$ is an affine function of the outcome $a_i$. Any feasible $r$ has the same normalization and mean as $q^{\ast }$, making the residual in the three-point identity for relative entropy zero.^[ref]

Let $p_i>0$ be a probability mass function on a finite alphabet with numerical outcomes $a_i$. Prescribe a mean $m$ strictly between the smallest and largest outcomes. Exponential tilting giveswith $\theta$ chosen so that ${\sum }_i{q}_i^{\ast }{a}_i=m$. For any other feasible law $r$, direct substitution yieldsRelative entropy nonnegativity therefore proves optimality and uniqueness. At an extreme prescribed mean, feasible laws may have zero probabilities; a finite tilting parameter need not realize them, although a limit can.

Take independent and identically distributed draws from the uniform three-outcome reference above. An observed sample mean at least $1/2$ is unusual because the source expected value is zero. For two draws, the only successful ordered pairs are $(0,1),(1,0),(1,1)$, giving probability $3/9=1/3$.

For long samples, the relevant empirical distributions have mean at least $1/2$. The method of types assigns a profile $q$ a leading probability cost $e^{-nD(q\Vert p)}$. The least costly admissible profile is the information projection on three outcomes: in this example the minimizing mean is on the boundary $1/2$, and the leading exponent is approximately $0.197378$ nats per draw. The exponential scale is not an exact finite-sample probability formula.^[ref]

The attraction of this connection is that a constrained optimization problem predicts the most probable way an unusual aggregate observation can occur. It links relative entropy, expected value constraints and rare empirical distributions without identifying a rare event with just one exact attainable type class.

In Information Theory and Statistics: A Tutorial by Imre Csiszár and Paul C. Shields (2004, Historical Notes, §3, printed p. 522), the authors connect systematic studies of information projection with earlier relative entropy minimization principles and geometric work, including Imre Csiszár's I-Divergence Geometry of Probability Distributions and Minimization Problems by Imre Csiszár (1975). This is the tutorial's historical account, rather than a claim of one isolated invention. Its finite-alphabet treatment makes the projection identities available without the additional machinery needed for general probability spaces.