Pinsker's inequality

With Total variation distance defined by $d_{\mathrm{TV}}(P,Q)={\sup }_A|P(A)-Q(A)|$ and relative entropy computed using natural logarithms, Pinsker's inequality statesIf $P$ is not absolutely continuous with respect to $Q$, the relative entropy is infinite and the bound is vacuous. A small relative entropy therefore forces small discrepancies in probabilities of all events. For example, $P=(0.6,0.4)$ and $Q=(0.5,0.5)$ have Total variation distance $0.1$ and relative entropy $0.6\log 1.2+0.4\log 0.8\simeq 0.02014$, giving an upper bound about $0.1003$.

A short proof reduces to a binary event. Grouping the probability mass inside and outside $A$, the log-sum inequality gives $D(P\Vert Q)\ge d(p\Vert q)$, with $p=P(A)$ and $q=Q(A)$. For $0<q<1$, the binary relative entropy as a function of $p$ has minimum zero at $p=q$ and second derivative $1/[p(1-p)]\ge 4$. Integrating this derivative bound twice gives $d(p\Vert q)\ge 2(p-q{)}^2$. Endpoint cases follow by limits. Taking the supremum over $A$ proves the inequality. Using base-two logarithms changes the constant by a factor of $\log 2$.