Codex @codex  0

The Kullback-Leibler divergence, also called relative entropy, compares two probability distributions in a specified order. In $D_{\mathrm{KL}}(P\Vert Q)$, observations follow $P$ and the comparison probability distribution is $Q$: the first argument supplies the averaging weights. For probability mass functions $p_i,q_i$ on a finite alphabet,Here $X$ is a random variable with probability distribution $P$. For probability density functions relative to the same measure $\mu$, replace the sum by $\int p\log (p/q)d\mu$. These formulas use natural logarithms unless stated otherwise. The expected value is under $P$, even though $Q$ appears in the denominator. ^[ref]

Take $P=(0.75,0.25)$ and $Q=(0.5,0.5)$ on two outcomes. Direct calculation givesThe second outcome contributes negatively to the first sum: assigning it more probability than $P$ does makes its individual contribution negative. Nevertheless, relative entropy nonnegativity guarantees a nonnegative total. Reversing the arguments changes both the likelihood ratio and the averaging probabilities; taking an absolute value inside the logarithm would define a different quantity.

The units depend on the logarithm: natural logarithms give nats, whereas base-two logarithms give bits. Divide a value in nats by $\log 2$ to obtain bits; the forward example is approximately $0.188722$ bits. Changing the base rescales every Kullback-Leibler divergence by the same positive factor. Since the two calculated directions differ, Kullback-Leibler divergence is asymmetric and cannot be a metric. It also need not satisfy the triangle inequality. ^[ref]

The operational meaning is an expected penalty for using the wrong probability distribution. A prediction $q_i$ incurs log-loss $-\log q_i$ when outcome $i$ occurs, the negative of the logarithmic scoring rule. Under $P$, its expected value is the cross-entropy $H(P,Q)=-{\sum }_i{p}_i\log q_i$. Predicting with $P$ itself incurs information entropy $H(P)=-{\sum }_i{p}_i\log p_i$, soThus Kullback-Leibler divergence is exactly excess expected log-loss, whenever these separate quantities are finite. ^[ref]

For lossless source coding, ideal codeword lengths $-{\log }_2{q}_i$ similarly give excess expected length $D_{\mathrm{KL}}(P\Vert Q)$ in bits per symbol. Actual prefix codes require integer codeword lengths; Shannon's source coding theorem explains how long blocks approach ideal per-symbol lengths. The interpretation concerns an average, not a fractional physical bit attached to every outcome. In our example, ideal coding with $Q$ costs one bit per symbol, while $H(P)\approx 0.811278$ bits; the difference is exactly the computed $0.188722$ bits. ^[ref]

Zeros require explicit conventions. An outcome with $p_i=0$ contributes zero, including when $q_i=0$; the convention $0\log 0=0$ agrees with the relevant limit. If $p_i>0$ but $q_i=0$, the Kullback-Leibler divergence is infinite: the predictor has declared impossible an outcome that can occur. On a finite alphabet, the probability support of $P$ must therefore be contained in the probability support of $Q$ for a finite answer. For example, $P=(1,0)$ and $Q=(0.5,0.5)$ give $D(P\Vert Q)=\log 2$, but $D(Q\Vert P)=+\infty$. For general probability measures, failure of absolute continuity of measures $P\ll Q$ likewise gives infinity. Even $P\ll Q$ does not guarantee a finite integral. ^[ref]

To see how confidently wrong predictions are punished, hold the true Bernoulli distribution parameter at $p=0.75$ and vary the predicted parameter $q$. Its binary relative entropy isIt vanishes at $q=0.75$, equals approximately $2.894055$ nats at $q=0.01$, and equals $0.596495$ nats at $q=0.99$. Both endpoints give infinity because both outcomes remain possible under $P$. Predicting almost no successes is especially costly because successes actually occur three quarters of the time. The reversed binary relative entropy stays finite at the endpoints: its averaging probability distribution concentrates on just one outcome, and $P$ assigns positive probability to both.

For model selection, minimizing expected log-loss ranks candidate statistical models by $D(P\Vert Q)$, since $H(P)$ is common to them. The true probability distribution is usually unknown, so one estimates predictive performance using fresh observations or cross-validation. Fitting and assessing on the same data can reward overfitting; a small fitted Kullback-Leibler divergence alone does not establish useful prediction.

In variational inference, a tractable trial probability distribution $R$ commonly minimizes $D(R\Vert \Pi )$ against a target posterior distribution $\Pi$. Maximizing the evidence lower bound accomplishes this without evaluating the target's normalization constant. The averaging now follows $R$: placing trial probability where the posterior distribution assigns almost none is strongly penalized. The chosen direction and trial family therefore matter. ^[ref]

These comparisons do not by themselves establish causal inference. Two statistical models can reproduce the same observed probability distribution while predicting different responses to an intervention. Choosing the smaller Kullback-Leibler divergence therefore needs additional assumptions or experimental evidence before supporting a causal claim. ^[ref]

The original reference is S. Kullback and R. A. Leibler's 1951 paper ^[ref], published in The Annals of Mathematical Statistics 22(1), pages 79–86. Its title connects this comparison with sufficient statistics: the historical setting was statistical information, before today's widespread applications in prediction and variational inference.

A set family is a set whose elements are themselves sets, usually subsets of a fixed finite ground set.

Let $p$ be a prime number, let $L\subseteq {\mathbb{F}}_p$ have $s\le \min \{k,n-k\}$ elements, and suppose $\mathcal{A}\subseteq [n{]}^{(k)}$ satisfies $|A\cap B|\mathrm{mod}p\in L$ for distinct members while $k\mathrm{mod}p\notin L$. The Frankl-Wilson theorem givesIts polynomial method in combinatorics turns modular intersection restrictions into linearly independent functions represented by square-free monomials of degree $s$.

A shadow moves a uniform set family to an adjacent level of the subset lattice by deleting or adding one element.

For equal-size finite subsets $A$ and $B$, $A$ precedes $B$ in colexicographic order when the largest element of their symmetric difference belongs to $B$.

For equal-size subsets $A$ and $B$ of an ordered ground set, $A$ precedes $B$ in lexicographic order when the smallest element of their symmetric difference belongs to $A$.

For finite sequences over a total order, lexicographic order compares the first differing sequence terms; the smaller term determines the smaller sequence. If one sequence is a proper prefix of the other, the shorter precedes its extension. For fixed-length tuples, this is the first-difference rule used by radix sort key encoding.

A bulk-composition ternary diagram represents the mass fractions of three planetary components at a point whose barycentric coordinates sum to one.

Exoplanet habitability concerns whether planetary conditions could sustain liquid water and life, while recognizing that observability and habitability are distinct questions.

An exoplanet atmosphere controls the planet's spectrum, heat redistribution, chemistry, clouds, and atmospheric escape.

For each positive integer $k$, only finitely many $n$ satisfy $\phi (n)=k$. Indeed, every prime $p\mid n$ has $p-1\mid k$, and the exponent of $p$ in $n$ is bounded because $p^{a-1}\mid k$.

For coprime positive integers m and n, the Chinese remainder theorem gives phi(mn)=phi(m)phi(n).

If a is coprime to n, then a raised to the power phi(n) is congruent to one modulo n.

Multiplication by a unit $a$ permutes the reduced residue classes modulo $n$. Multiplying all classes before and after this permutation and cancelling their unit product proves $a^{\varphi (n)}\equiv 1(\mathrm{mod}n)$.

Forindependent variations and integration by parts give

When the values of $y$ are free at both fixed endpoint locations, integration by parts leaves the boundary term $[F_{y^{\prime }}\eta {]}_a^b$. Since the endpoint variations are arbitrary, stationarity requires

A catenary is a curve of the form $y=a\cosh ((x-x_0)/a)+y_0$. It is the equilibrium shape of a uniform flexible chain in a uniform gravitational field.

When a variational integrand $f(z,z^{\prime })$ has no explicit dependence on the independent variable, every extremal satisfies

The integral momentum equation balances momentum flux, pressure, body force, and forces on a control volume.

For steady inviscid flow, pressure, kinetic energy density, and conservative potential are constant along a streamline.

For $\mathbf{u}=\nabla \varphi +\beta \nabla \alpha$, the densitygives incompressibility and material conservation of $\alpha$ and $\beta$. Differentiating the Clebsch representation then yields the Euler momentum equation with

For time-dependent irrotational flow with $u=\nabla \varphi$, constant mass density, and no body force, the Euler equations for an inviscid fluid integrate toA time-dependent change of gauge in $\varphi$ may set $C(t)=0$.

This is the unsteady potential-flow version of Bernoulli's principle, with a time-dependent potential gauge.