f-divergence ($D_f(P\Vert Q)$)

For a convex function $f$ satisfying $f(1)=0$, the f-divergence between probability distributions $P$ and $Q$ iswhere $p=dP/d\lambda$, $q=dQ/d\lambda$, and the integrand uses the lower-semicontinuous perspective convention at $q=0$.

For probability densities $p$ and $q$, the Kullback-Leibler divergence from $q$ to $p$ is

For probability distributions $p$ and $q$ on the same finite set,with equality exactly when $p=q$ on the support of $p$.

For probability distributions $P$ and $Q$ on a product space, the Kullback-Leibler divergence decomposes into the divergence of a marginal and the expected divergence of the corresponding conditional distributions. Iteration gives

Passing two probability distributions through the same Markov kernel cannot increase their relative entropy. For a kernel $K$,

For nonnegative $a_i,b_i$, with $a={\sum }_i{a}_i$ and $b={\sum }_i{b}_i$, the log-sum inequality isIt proves both Gibbs inequality and the data processing inequality for relative entropy.

For a probability mass function $Q$ of full support on a finite alphabet and a real function $g$,where $D_e$ uses natural logarithms. The unique maximizing probability mass function is the exponential tilt $P^{\ast }(x)=Q(x)e^{g(x)}/{\mathbb{E}}_Q{e}^g$.

For univariate normal distributions,

For probability mass functions $P,Q$ with $P$ absolutely continuous with respect to $Q$,

For a measurable function $T$, every f-divergence contracts under pushforward:This follows from conditional Jensen inequality applied to the jointly convex perspective $(a,b)\mapsto bf(a/b)$.

The squared Hellinger distance iswhere $p$ and $q$ are densities with respect to any common dominating measure $\lambda$.