= Pinsker's inequality
{c}
{wiki}
With <Total variation distance> defined by $d_{\mathrm{TV}}(P,Q)=\sup_A|P(A)-Q(A)|$ and <Kullback-Leibler divergence>[relative entropy] computed using <natural logarithm>[natural logarithms], <Pinsker's inequality> states
$$
d_{\mathrm{TV}}(P,Q)\leq\sqrt{\tfrac12D(P\Vert Q)}.
$$
If $P$ is not absolutely continuous with respect to $Q$, the <Kullback-Leibler divergence>[relative entropy] is infinite and the bound is vacuous. A small <Kullback-Leibler divergence>[relative entropy] therefore forces small discrepancies in <probability>[probabilities] of all <events>. For example, $P=(0.6,0.4)$ and $Q=(0.5,0.5)$ have <Total variation distance> $0.1$ and <Kullback-Leibler divergence>[relative entropy] $0.6\log1.2+0.4\log0.8\simeq0.02014$, giving an upper bound about $0.1003$.
A short proof reduces to a <binary variable>[binary] <event>. Grouping the <probability mass> inside and outside $A$, the <log-sum inequality> gives $D(P\Vert Q)\geq d(p\Vert q)$, with $p=P(A)$ and $q=Q(A)$. For $0<q<1$, the <binary variable>[binary] <Kullback-Leibler divergence>[relative entropy] as a <function> of $p$ has minimum zero at $p=q$ and <second derivative> $1/[p(1-p)]\geq4$. Integrating this <derivative> bound twice gives $d(p\Vert q)\geq2(p-q)^2$. Endpoint cases follow by limits. Taking the supremum over $A$ proves the inequality. Using base-two <logarithms> changes the constant by a factor of $\log2$.
Back to article page
