Kullback-Leibler divergence ($D_{\mathrm{KL}}$)

For probability densities $p$ and $q$, the Kullback-Leibler divergence from $q$ to $p$ is

For a measurable event $A$, applying Jensen inequality to $u\log u$ separately on $A$ and its complement shows that the Kullback-Leibler divergence between two probability measures is at least the divergence between the two event indicators. This reduces a general probability measure comparison to a binary one and is the key step in a proof of Pinsker's inequality.

For $d(a,b)=a\log (a/b)+(1-a)\log ((1-a)/(1-b))$, the second derivative with respect to $a$ is $1/[a(1-a)]\ge 4$. Its value and first derivative at $a=b$ vanish, so $d(a,b)\ge 2(a-b{)}^2$. Boundary values follow by continuity or the infinite-divergence convention. Together with the binary partition bound for relative entropy, this gives Pinsker's inequality with the natural-logarithm constant.

For sufficiently regular positive probability densities, this compares their spatial logarithmic derivatives. It is the dissipation functional for relative entropy under suitable diffusion equations. For standard Gaussian $g=\gamma$, it becomes $\int |\nabla \log f+v{|}^{2}f$, and in one dimension equals $\int |f^{\prime }{|}^2/f+\int v^{2}f-2$ at unit mass.

The differential estimate $I^{\prime }(t)\le -2I(t)$ gives the displayed exponential bound by an integrating factor. Start at any time with finite information. Combining this with a zero limiting entropy yields $H(f_t\mid \gamma )={\int }_t^{\infty }I(s)ds\le I(t)/2$.

For probability densities $f,g$ with $g>0$, set $q=f/g$. Equal masses give $H(f\mid g)=\int g(q\log q-q+1)\ge 0$. The scalar integrand is minimized only at $q=1$, so equality means $f=g$ almost everywhere. Use $0\log 0=0$; infinite entropy is allowed.

For nonnegative densities of possibly different total masses, this divergence includes the mass correction $-x+y$ as well as the logarithmic term. It is the Bregman distance of the maximum-entropy regularization functional at a positive reference $y$. Convexity of $s\log s-s$ proves nonnegativity. For equal masses the correction integrates to zero, recovering the Kullback-Leibler divergence; for probability density functions both masses are one. Use the usual extended-value conventions at zero reference density.

For probability mass functions $P,Q,R$ with compatible supports,When the final term vanishes, this is the Pythagorean identity for relative entropy.

The Pythagorean identity iswhenever the residual term in the three-point identity for relative entropy is zero. Convex information projections replace equality by the corresponding Pythagorean inequality.

The binary relative entropy is the Kullback-Leibler divergence between Bernoulli distributions:It is the exact exponent obtained by optimizing the Chernoff bound for a Bernoulli mean.

For probability distributions $p$ and $q$ on the same finite set,with equality exactly when $p=q$ on the support of $p$.

For probability distributions $P$ and $Q$ on a product space, the Kullback-Leibler divergence decomposes into the divergence of a marginal and the expected divergence of the corresponding conditional distributions. Iteration gives

Passing two probability distributions through the same Markov kernel cannot increase their relative entropy. For a kernel $K$,

For nonnegative $a_i,b_i$, with $a={\sum }_i{a}_i$ and $b={\sum }_i{b}_i$, the log-sum inequality isIt proves both Gibbs inequality and the data processing inequality for relative entropy.

For a probability mass function $Q$ of full support on a finite alphabet and a real function $g$,where $D_e$ uses natural logarithms. The unique maximizing probability mass function is the exponential tilt $P^{\ast }(x)=Q(x)e^{g(x)}/{\mathbb{E}}_Q{e}^g$.

For univariate normal distributions,

If $P={⨂}_{i=1}^n{P}_i$ is a product measure, $Q$ is any probability measure, and $X^{(i)}$ omits coordinate $i$, then

For a random vector $X_{1:n}$,This is equivalent to Han's inequality for relative entropy after expansion against a product reference measure.

The strong formfollows by iterating entropy submodularity.

For a product reference $P={⨂}_i{P}_i$ and arbitrary $Q$,It follows from the chain rule for relative entropy and convexity of relative entropy in its first argument.