= Solution
Use a fixed <logarithm> base $b>1$, and write $c=\log_b e=1/\ln b$. All <relative entropies> below use this same base. Adopt $0\log(0/q)=0$ and $p\log(p/0)=+\infty$ for $p>0$.
For <Gibbs inequality>, if $q(x)=0<p(x)$ at some point, the <relative entropy> is infinite and the required conclusion is immediate. Otherwise let $S=\{x:p(x)>0\}$. The elementary inequality $\ln u\leq u-1$, with equality only at $u=1$, gives, for every $x\in S$,
$$
p(x)\log_b\frac{p(x)}{q(x)}\geq c\bigl(p(x)-q(x)\bigr).
$$
Consequently
$$
D(p\Vert q)\geq c\left(1-\sum_{x\in S}q(x)\right)\geq0.
$$
This argument applies to a countable <discrete probability distribution> too: the difference between the two sides of the pointwise inequality is nonnegative, and the negative part of the entropy summand is bounded by $cq(x)$, hence summable. Thus no subtraction of divergent series is involved. Equality forces both $\sum_Sq=1$ and equality in every pointwise inequality; hence $q(x)=p(x)$ on $S$ and $q=0=p$ off $S$. Conversely $p=q$ plainly gives zero. Therefore $\boxed{D(p\Vert q)\geq0,\quad D(p\Vert q)=0\iff p=q}$.
For the <log-sum inequality>, set $F=\sum_Af$, $G=\sum_Ag$, and normalize to the <probability distributions> $p_A=f/F$, $q_A=g/G$. When $A$ is nonempty, positivity ensures $F,G>0$, and direct expansion gives
$$
\sum_{x\in A}f(x)\log_b\frac{f(x)}{g(x)}
=F D(p_A\Vert q_A)+F\log_b\frac FG
\geq F\log_b\frac FG.
$$
Thus \b[normalization reduces the log-sum inequality to Gibbs inequality]. Equality holds exactly when $f/F=g/G$. The empty-set case is the zero identity under the usual zero-mass convention.
For the final bound, partition the alphabet using $A=\{x:p(x)>q(x)\}$ and put $a=p(A)$, $d=q(A)$. Since the total signed difference is zero,
$$
a-d=\sum_{x\in A}(p(x)-q(x))
=\frac12\sum_x|p(x)-q(x)|.
$$
Apply the <log-sum inequality> separately to $A$ and its complement. The same normalization argument works for countable sets of finite total mass, using the countable <Gibbs inequality> just proved. It yields the <binary partition bound for relative entropy>
$$
D(p\Vert q)\geq a\log_b\frac ad+(1-a)\log_b\frac{1-a}{1-d}.
$$
Zero block masses are interpreted by the stated conventions; a positive mass divided by zero makes the bound infinite. The given binary estimate now implies
$$
D(p\Vert q)\geq2c(a-d)^2
=\boxed{\frac{\log_b e}{2}\left(\sum_x|p(x)-q(x)|\right)^2}.
$$
This is <Pinsker's inequality>. Equivalently, for the <total variation distance> $\|p-q\|_{\mathrm{TV}}=\frac12\sum_x|p(x)-q(x)|$, the bound is $D(p\Vert q)\geq2\log_b e\,\|p-q\|_{\mathrm{TV}}^2$. The factor $\log_b e$ keeps the statement valid for either bits or natural-logarithm units.
Back to article page
