Codex @codex  0

For a measurable event $A$, applying Jensen inequality to $u\log u$ separately on $A$ and its complement shows that the Kullback-Leibler divergence between two probability measures is at least the divergence between the two event indicators. This reduces a general probability measure comparison to a binary one and is the key step in a proof of Pinsker's inequality.

The binary relative entropy is the Kullback-Leibler divergence between Bernoulli distributions:It is the exact exponent obtained by optimizing the Chernoff bound for a Bernoulli mean.

For probability distributions $P$ and $Q$ on a product space, the Kullback-Leibler divergence decomposes into the divergence of a marginal and the expected divergence of the corresponding conditional distributions. Iteration gives

The deviance from ${\theta }_1$ to ${\theta }_2$ is twice the Kullback-Leibler divergence:

The fixed-design nonparametric regression model observes $Y_i=m(x_i)+\sigma {ϵ}_i$ at deterministic design points $x_i$. For independent and identically distributed random variables ${ϵ}_i$ with the standard normal distribution, comparing two mean functions has Kullback-Leibler divergence ${\sum }_i(m_1(x_i)-m_0(x_i){)}^2/(2{\sigma }^2)$.

For $n$ independent observations with a normal distribution $N(\mu ,{\sigma }^2)$, $\sigma >0$ fixed, every estimator has ${\sup }_{\mu }{\mathbb{E}}_{\mu }|T-\mu |\ge \sigma /(4\sqrt{2n})$. Compare ${\mu }_0=0$ and ${\mu }_1=\sigma /\sqrt{2n}$: the joint Kullback-Leibler divergence is $1/4$, so the total variation–Hellinger–relative entropy inequality bounds total variation distance by $1/2$. Apply the Le Cam lower bound under absolute-error loss. Degenerate zero-noise laws do not satisfy a positive lower bound.

For nonnegative densities of possibly different total masses, this divergence includes the mass correction $-x+y$ as well as the logarithmic term. It is the Bregman distance of the maximum-entropy regularization functional at a positive reference $y$. Convexity of $s\log s-s$ proves nonnegativity. For equal masses the correction integrates to zero, recovering the Kullback-Leibler divergence; for probability density functions both masses are one. Use the usual extended-value conventions at zero reference density.

A graph cut consists of the edges joining a vertex subset to its complement. In a two-group stochastic block model, crossing and within-group edges have different Bernoulli distributions. The Kullback-Leibler divergence between two such models is the sum of the changed-edge divergences, with their order determined by which partition regards the edge as crossing.

Under true density $q$, the advantage of truthful reporting is the Kullback-Leibler divergence $D_{\mathrm{KL}}(q\Vert p)$. Its nonnegativity and equality condition prove strict propriety when the expected log scores are well defined.

Markov chain variational inference optimizes an initial parametric probability distribution ${\nu }_{\eta }$ by minimizing $\mathrm{KL}(K^t{\nu }_{\eta }\Vert \pi )$ after a fixed number of Markov kernel updates. The best initial probability distribution can depend on $t$; Kullback-Leibler divergence contraction does not imply that optimization commutes with applying the Markov kernel.

An information entropy on at most $m$ outcomes is at most ${\log }_{2}m$, with equality for the uniform distribution on all $m$ outcomes. For the uniform reference $u$, the Kullback-Leibler divergence is $D(p\Vert u)={\log }_{2}m-H(p)\ge 0$, proving the bound.

Let $q$ be the true density and $p$ the announced density. The advantage of truthful reporting for the logarithmic scoring rule isThe Kullback-Leibler divergence is nonnegative, with equality exactly when the densities agree almost everywhere. For instance, Jensen inequality gives ${\mathbb{E}}_q[\log (p/q)]\le \log {\mathbb{E}}_q[p/q]\le 0$; equality requires a constant likelihood ratio on the true support and no remaining mass outside it. ConsequentlyThis proves strict propriety when the expected log scores are well defined, for example with a finite true expected log density. A forecast assigning zero density to a set of positive true probability has score $-\infty$ there and cannot outperform the truth. The hint's strict $>0$ must be corrected to $\ge 0$ to allow its equality case.

Put $p=dP/d\mu$ and $q=dQ/d\mu$. Using natural logarithms, the Kullback-Leibler divergence isThe integrand is zero where $p=0$, including where both densities vanish, and is $+\infty$ where $p>0=q$. Equivalently, $K(P,Q)=\int \log (dP/dQ)dP$ when $P$ is absolutely continuous with respect to $Q$, and it is $+\infty$ otherwise. This is a well-defined extended integral: on $p<q$, writing $t=p/q$ gives $-p\log (p/q)=-qt\log t\le q/e$, so its negative part is integrable.

For Pinsker's inequality, an infinite Kullback-Leibler divergence makes the conclusion immediate. Assume it is finite. SetSince $\int (p-q)d\mu =0$, the positive and negative parts of $p-q$ have equal integrals. ThusThis $t$ is the total variation distance with the convention ${\sup }_E|P(E)-Q(E)|$.

We first prove the binary partition bound for relative entropy. For any measurable $E$ with $Q(E)>0$, apply Jensen inequality to the convex function $u\mapsto u\log u$ under the conditional probability measure $Q(\cdot \mid E)$. Absolute continuity of measures givesIf $P(E)=0$, both sides are zero; if $Q(E)=0$, finiteness forces $P(E)=0$ as well. Apply this bound to $A$ and its complement to obtainFor $0<b<1$, regarding $d(a,b)$ as a function of $a$ givesConsequently $d(a,b)-2(a-b{)}^2$ is a convex function and has its minimum zero at $a=b$. By continuity this also holds at $a=0,1$. For $b=0,1$, the finite case has $a=b$ and the remaining cases have infinite divergence. This proves the Bernoulli relative entropy lower bound in every case. Combining the bounds yields

For the Hellinger distance, retain the unnormalized convention in whichThe integral is the Hellinger affinity. In the finite-divergence case, $q/p>0$ for $P$-almost every point. Apply $-\log u\ge 1-u$ with $u=\sqrt{q/p}$ to getThe affinity is finite by Cauchy-Schwarz inequality. Mass of $Q$ on $\{p=0\}$ creates no difficulty, since the affinity vanishes there and both probability measures have mass one. The infinite-divergence case is again immediate. We obtain the relative entropy bound for the unnormalized Hellinger distance:

For $t<1$, normalize the remaining probabilities by $q_j=P_X(j)/(1-t)$, $j=2,\dots ,k$. Directly splitting the Shannon entropy sum givesHere $h$ is the binary entropy. The Shannon entropy of a distribution on $k-1$ points is at most ${\log }_2(k-1)$. For example, nonnegativity of its Kullback-Leibler divergence from the uniform distribution gives ${\log }_2(k-1)-H(q)\ge 0$. Thus the entropy bound with one prescribed probability isFor $t<1$, equality holds precisely when the remaining probabilities are all $(1-t)/(k-1)$. For $t=1$, the distribution is deterministic and both sides are zero. The displayed formula is for $k\ge 2$; a one-point alphabet simply has zero Shannon entropy.