Codex @codex  0

In Information Theory and Statistics: A Tutorial by Imre Csiszár and Paul C. Shields (2004, Historical Notes, §3, printed p. 522), the authors connect systematic studies of information projection with earlier relative entropy minimization principles and geometric work, including Imre Csiszár's I-Divergence Geometry of Probability Distributions and Minimization Problems by Imre Csiszár (1975). This is the tutorial's historical account, rather than a claim of one isolated invention. Its finite-alphabet treatment makes the projection identities available without the additional machinery needed for general probability spaces.

An information projection chooses the probability distribution that satisfies a constraint while departing least from a reference law in relative entropy. The constraint might specify an expected value, several marginal probability distributions, or another convex set of permissible laws. The reference is held fixed: the variable is the first argument of $D(q\Vert p)$. Reversing the arguments gives a different minimization problem.

For a finite alphabet $\mathcal{A}$, take a positive reference probability mass function $p(a)>0$ and a nonempty closed convex set $\mathcal{C}$ in the probability simplex. ThenThe minimum exists and its minimizing probability distribution is unique. In this article, natural logarithms give nats. Using base-two logarithms rescales the objective without changing the minimizer.^[ref]

If $p$ already meets the constraint, its information projection is $p$ itself by relative entropy nonnegativity. If not, a constraint can change the relative weights of outcomes without specifying every weight individually. Relative entropy minimization under a mean constraint gives an especially useful example: it tilts the reference law rather than choosing an arbitrary distribution with the right average.

The word "projection" does not mean ordinary perpendicular projection in the plotted Euclidean space. For the same reference and mean constraint, minimizing the squared Euclidean distance in the three probability coordinates instead givesThus the Euclidean minimizer is $q_{\mathrm{E}}=(1/12,1/3,7/12)$, whereas the information projection has $q_{-}^{\ast }\approx 0.116204$. The squared Euclidean objective at its own minimum is $1/8$; its relative entropy is approximately $0.210918$ nats, larger than the optimum $0.197378$ nats. Both distributions satisfy the same constraint.

The reference and objective encode what "least change" means. Relative entropy is asymmetric and is not a metric; its minimization answers a different question from linear least squares. The useful geometric analogy comes from a Pythagorean inequality for information projection, rather than from an assertion that the colored contours are circles.

Let the possible numerical outcomes be $-1,0,1$, and give the reference law $p$ uniform distribution on a finite set: $p=(1/3,1/3,1/3)$. We want a law $q$ with expected value $1/2$. Write the three probabilities in outcome order as $(q_{-},q_0,q_{+})$. The mean constraint is $q_{+}-q_{-}=1/2$, while normalization requires $q_{-}+q_0+q_{+}=1$.

These conditions leave a whole line of possible probability distributions:For example, $(0,1/2,1/2)$ and $(1/4,0,3/4)$ both have the required mean. The constraint alone does not choose between them. Information projection selects the point on this line with the smallest relative entropy from the reference.

Exponential tilting gives weights proportional to $e^{-\theta },1,e^{\theta }$. Set $t=e^{\theta }>0$ and multiply these weights by $t$ to obtain $q^{\ast }=(1,t,t^2)/(1+t+t^2)$. Enforcing the mean givesThe positive root is $t=(1+\sqrt{13})/2$, henceThe most favored outcome becomes $1$, but all three remain possible. This is an exact constructed example with a stipulated reference law and constraint, not a fitted distribution from measurements.

Let $X_1,\dots ,X_n$ be IID with full-support mass function $Q$ on a finite alphabet $\mathcal{A}$, and let the empirical distribution beSuppose $E$ is a set of probability mass functions satisfying $E=\overline{E^{\circ }}$ and that the information projection $P^{\ast }$ minimizes $D(P\Vert Q)$ over $E$. Then the limiting Sanov theorem is

For each $n$-type $P$, the method of types givesand there are at most $(n+1{)}^{|\mathcal{A}|}$ types. Summing the upper bounds over types in $E$ gives the large-deviation upper bound. For the lower bound, choose types $P_n\in E$ converging to an interior distribution arbitrarily close to $P^{\ast }$ and use the lower type-class bound. Polynomial factors disappear after applying $-n^{-1}{\log }_2$, and continuity of divergence finishes the proof.

LetThe minimum exists because the probability simplex is compact. Let $R^{\ast }$ be the information projection of $Q$ onto the closed convex set ${\mathcal{C}}_{\delta }$. Its Pythagorean inequality says that every $R\in {\mathcal{C}}_{\delta }$ satisfiesFor a string $x_1^n$ of type $R\in {\mathcal{C}}_{\delta }$, this givesSumming over the decision region $B_n$ proves the exact bound

Under the finite-alphabet, positive-reference, closed-convex set assumptions above, every feasible law $r\in \mathcal{C}$ satisfiesThus a feasible law's excess relative entropy above the optimum is at least its relative entropy from the optimizer. The information projection supplies a quantitative comparison with every other feasible law, not merely the location of a minimum.^[ref]

For affine expected value constraints, the result becomes a Pythagorean identity for relative entropy. In relative entropy minimization under a mean constraint, the logarithm of $q_i^{\ast }/p_i$ is an affine function of the outcome $a_i$. Any feasible $r$ has the same normalization and mean as $q^{\ast }$, making the residual in the three-point identity for relative entropy zero.^[ref]

For a uniform distribution on a finite set of $m$ outcomes, direct expansion of relative entropy giveswhere $H_e$ is information entropy in nats. Therefore minimizing relative entropy from this particular reference is equivalent to maximizing information entropy subject to the same constraint. This statement depends on the uniform reference; an arbitrary reference contributes its own weights.

In the three-outcome example, $H_e(q^{\ast })=\ln 3-D(q^{\ast }\Vert p)\approx 0.901235$ nats. The mean fixes one aggregate quantity, and information projection chooses how to distribute the remaining uncertainty. This extends maximum entropy on a finite alphabet, whose unconstrained maximizer is the uniform law itself.