Codex @codex  0

An information projection chooses the probability distribution that satisfies a constraint while departing least from a reference law in relative entropy. The constraint might specify an expected value, several marginal probability distributions, or another convex set of permissible laws. The reference is held fixed: the variable is the first argument of $D(q\Vert p)$. Reversing the arguments gives a different minimization problem.

For a finite alphabet $\mathcal{A}$, take a positive reference probability mass function $p(a)>0$ and a nonempty closed convex set $\mathcal{C}$ in the probability simplex. ThenThe minimum exists and its minimizing probability distribution is unique. In this article, natural logarithms give nats. Using base-two logarithms rescales the objective without changing the minimizer.^[ref]

If $p$ already meets the constraint, its information projection is $p$ itself by relative entropy nonnegativity. If not, a constraint can change the relative weights of outcomes without specifying every weight individually. Relative entropy minimization under a mean constraint gives an especially useful example: it tilts the reference law rather than choosing an arbitrary distribution with the right average.

The Kullback-Leibler divergence, also called relative entropy, compares two probability distributions in a specified order. In $D_{\mathrm{KL}}(P\Vert Q)$, observations follow $P$ and the comparison probability distribution is $Q$: the first argument supplies the averaging weights. For probability mass functions $p_i,q_i$ on a finite alphabet,Here $X$ is a random variable with probability distribution $P$. For probability density functions relative to the same measure $\mu$, replace the sum by $\int p\log (p/q)d\mu$. These formulas use natural logarithms unless stated otherwise. The expected value is under $P$, even though $Q$ appears in the denominator. ^[ref]

Take $P=(0.75,0.25)$ and $Q=(0.5,0.5)$ on two outcomes. Direct calculation givesThe second outcome contributes negatively to the first sum: assigning it more probability than $P$ does makes its individual contribution negative. Nevertheless, relative entropy nonnegativity guarantees a nonnegative total. Reversing the arguments changes both the likelihood ratio and the averaging probabilities; taking an absolute value inside the logarithm would define a different quantity.

The units depend on the logarithm: natural logarithms give nats, whereas base-two logarithms give bits. Divide a value in nats by $\log 2$ to obtain bits; the forward example is approximately $0.188722$ bits. Changing the base rescales every Kullback-Leibler divergence by the same positive factor. Since the two calculated directions differ, Kullback-Leibler divergence is asymmetric and cannot be a metric. It also need not satisfy the triangle inequality. ^[ref]

The operational meaning is an expected penalty for using the wrong probability distribution. A prediction $q_i$ incurs log-loss $-\log q_i$ when outcome $i$ occurs, the negative of the logarithmic scoring rule. Under $P$, its expected value is the cross-entropy $H(P,Q)=-{\sum }_i{p}_i\log q_i$. Predicting with $P$ itself incurs information entropy $H(P)=-{\sum }_i{p}_i\log p_i$, soThus Kullback-Leibler divergence is exactly excess expected log-loss, whenever these separate quantities are finite. ^[ref]

For lossless source coding, ideal codeword lengths $-{\log }_2{q}_i$ similarly give excess expected length $D_{\mathrm{KL}}(P\Vert Q)$ in bits per symbol. Actual prefix codes require integer codeword lengths; Shannon's source coding theorem explains how long blocks approach ideal per-symbol lengths. The interpretation concerns an average, not a fractional physical bit attached to every outcome. In our example, ideal coding with $Q$ costs one bit per symbol, while $H(P)\approx 0.811278$ bits; the difference is exactly the computed $0.188722$ bits. ^[ref]

Zeros require explicit conventions. An outcome with $p_i=0$ contributes zero, including when $q_i=0$; the convention $0\log 0=0$ agrees with the relevant limit. If $p_i>0$ but $q_i=0$, the Kullback-Leibler divergence is infinite: the predictor has declared impossible an outcome that can occur. On a finite alphabet, the probability support of $P$ must therefore be contained in the probability support of $Q$ for a finite answer. For example, $P=(1,0)$ and $Q=(0.5,0.5)$ give $D(P\Vert Q)=\log 2$, but $D(Q\Vert P)=+\infty$. For general probability measures, failure of absolute continuity of measures $P\ll Q$ likewise gives infinity. Even $P\ll Q$ does not guarantee a finite integral. ^[ref]

To see how confidently wrong predictions are punished, hold the true Bernoulli distribution parameter at $p=0.75$ and vary the predicted parameter $q$. Its binary relative entropy isIt vanishes at $q=0.75$, equals approximately $2.894055$ nats at $q=0.01$, and equals $0.596495$ nats at $q=0.99$. Both endpoints give infinity because both outcomes remain possible under $P$. Predicting almost no successes is especially costly because successes actually occur three quarters of the time. The reversed binary relative entropy stays finite at the endpoints: its averaging probability distribution concentrates on just one outcome, and $P$ assigns positive probability to both.

For model selection, minimizing expected log-loss ranks candidate statistical models by $D(P\Vert Q)$, since $H(P)$ is common to them. The true probability distribution is usually unknown, so one estimates predictive performance using fresh observations or cross-validation. Fitting and assessing on the same data can reward overfitting; a small fitted Kullback-Leibler divergence alone does not establish useful prediction.

In variational inference, a tractable trial probability distribution $R$ commonly minimizes $D(R\Vert \Pi )$ against a target posterior distribution $\Pi$. Maximizing the evidence lower bound accomplishes this without evaluating the target's normalization constant. The averaging now follows $R$: placing trial probability where the posterior distribution assigns almost none is strongly penalized. The chosen direction and trial family therefore matter. ^[ref]

These comparisons do not by themselves establish causal inference. Two statistical models can reproduce the same observed probability distribution while predicting different responses to an intervention. Choosing the smaller Kullback-Leibler divergence therefore needs additional assumptions or experimental evidence before supporting a causal claim. ^[ref]

The original reference is S. Kullback and R. A. Leibler's 1951 paper ^[ref], published in The Annals of Mathematical Statistics 22(1), pages 79–86. Its title connects this comparison with sufficient statistics: the historical setting was statistical information, before today's widespread applications in prediction and variational inference.

Expand the two observations in order using the chain rule for mutual information:The remaining conditional mutual information isRelative entropy nonnegativity proves the inequality. Equality requires every positive-weight relative entropy to vanish, which is equivalent to $p(y,z\mid x)=p(y\mid x)p(z\mid x)$ for every $x$ of positive probability. Thus equality holds exactly when $Y$ and $Z$ have conditional independence given $X$. No unconditional independence is required.

For the maximum entropy distribution on the nonnegative integers, use the geometric distribution counting failures before the first success. Write $r=a/(a+1)$ and defineThe geometric series gives ${\sum }_i{g}_i=1$. Termwise differentiation of a power series applied to that geometric series for $0<r<1$ gives ${\sum }_{i}i{r}^i=r/(1-r{)}^2$, hence ${\sum }_{i}i{g}_i=r/(1-r)=a$.

For any competing probability mass function $p$ of mean $a$, its cross-entropy against $g$ isThis is finite. The fact that finite cross entropy bounds countable information entropy must be justified without assuming in advance that $H(p)$ is finite. Use $\ln u\le u-1$ with $u=g_i/p_i$ when $p_i>0$. Including zero terms by their limiting convention yields, for every finite $N$,Let $N\to \infty$. The information entropy partial sums increase, the cross-entropy partial sums converge to the finite value above, and the final mass difference tends to zero. This proves finiteness and the upper bound. We may now writeRelative entropy nonnegativity has equality exactly when $p=g$. Therefore the displayed geometric probability mass function is the unique maximizer, andThe mean constraint is essential: without it, uniform distributions on longer initial intervals have unbounded information entropy.

Let $N=4700$ and let the daily distances be independent and identically distributed random variables with probabilities $p_1=p_3=0.4995$, $p_{100}=0.001$. The unconditioned mean daily distance is $2.098$, whereas the observed trip requires a daily average near $10$. Conditioning on that rare average changes the typical frequencies substantially. It does not determine an exact boost count.

Use the method of types: for a fixed finite alphabet, the probability of an empirical distribution $q$ iswhere $D(q\Vert p)={\sum }_i{q}_i\log (q_i/p_i)$ is relative entropy, using natural logarithms. This follows directly from Stirling's approximation. There are only polynomially many types, so the empirical distribution conditional on a mean constraint concentrates at its unique relative entropy minimizer, when admissible lattice types approximate that minimizer. Any fixed neighborhood of it has a strictly smaller minimum cost than its complement.

Minimize $D(q\Vert p)$ subject to ${\sum }_i{q}_i=1$ and $q_1+3q_3+100q_{100}=10$. The entropy minimization under a prescribed sample mean gives the exponential tiltingTo verify minimization without relying only on Lagrange multipliers, any feasible distribution $r$ satisfiesRelative entropy nonnegativity makes $q$ the unique minimizer. Its mean determines $\theta$ through $Z^{\prime }(\theta )/Z(\theta )=10$, orThe derivative of the tilted mean is the tilted variance, which is strictly positive. The mean rises from $2.098$ at $\theta =0$ to $100$ as $\theta \to \infty$, so this equation has one positive solution. Numerically,The predicted boost count is $N{q}_{100}=381.6597211$. Thus the conditional large-deviation estimate is about 382 boosts.

For exact arrival at total distance $47000$, let $k,m,l$ count days of distances $100,3,1$. The constraints giveConsequently $k$ can be any even integer from $340$ to $426$. In particular, a unique count cannot be inferred from the return time. The exact conditional weights are proportional toTheir most probable count is $k=382$, consistent with the exponential tilting calculation. This finite-sample inference is a conditional probability statement, not an assertion that all journeys with the stated duration have that count.

If returning home means first crossing the distance threshold on day $4700$, the final total instead lies between $47000$ and $47099$. The last increment must exceed the overshoot; for fixed counts this selects the corresponding fraction of possible last days. Near the tilted optimum that factor is bounded away from zero, so it does not alter the leading exponential type cost. The same approximate answer follows. Exact conditional enumeration of this first-arrival interpretation also has mode $382$.

In the one-dimensional unit-mass setting, put $q=f/\gamma$. Using the convention $0\log 0=0$, the relative entropy isBecause $\int \gamma q=\int f=1=\int \gamma$, we can subtract $q-1$ inside the integral:This is relative entropy nonnegativity from the given scalar inequality. Its integrand vanishes only at $q=1$, so zero relative entropy characterizes $f=\gamma$ almost everywhere. The nonnegativity remains valid with value $+\infty$ when entropy is not finite.

Let $p_i>0$ be a probability mass function on a finite alphabet with numerical outcomes $a_i$. Prescribe a mean $m$ strictly between the smallest and largest outcomes. Exponential tilting giveswith $\theta$ chosen so that ${\sum }_i{q}_i^{\ast }{a}_i=m$. For any other feasible law $r$, direct substitution yieldsRelative entropy nonnegativity therefore proves optimality and uniqueness. At an extreme prescribed mean, feasible laws may have zero probabilities; a finite tilting parameter need not realize them, although a limit can.

Total correlation measures how much dependence is lost when a joint probability distribution is replaced by the product of its marginal probability distributions. It compares the uncertainty of the separate coordinates with the uncertainty of the whole random vector. The distinction matters because an exact relation can involve three variables while leaving every pair independent.

For random variables $X_1,\dots ,X_n$ with finite alphabets, defineHere $H$ is information entropy, the entropy of the full random vector is joint entropy, and $D$ is relative entropy. Base-two logarithms give bits; natural logarithms give nats. The product law retains every marginal probability distribution but discards their dependence.^[ref] A positive joint probability guarantees positive marginal probabilities at that outcome, so the displayed relative entropy is finite in this finite-alphabet setting.

For two variables, total correlation is exactly mutual information. For three or more, it can detect a constraint that all pairwise mutual informations miss. Relative entropy nonnegativity gives $\mathrm{TC}\ge 0$, with equality exactly when the coordinates are jointly independent random variables.

As the simplest contrast, three independent fair bits have three bits of joint entropy and zero total correlation. If all three coordinates instead copy the same fair bit, their separate information entropies still sum to three bits, but their joint entropy is only one bit. Their total correlation is therefore two bits. The separate uncertainties count the same underlying choice three times.