 Solution

Use a fixed logarithm base $b>1$, and write $c={\log }_{b}e=1/\ln b$. All relative entropies below use this same base. Adopt $0\log (0/q)=0$ and $p\log (p/0)=+\infty$ for $p>0$.

For Gibbs inequality, if $q(x)=0<p(x)$ at some point, the relative entropy is infinite and the required conclusion is immediate. Otherwise let $S=\{x:p(x)>0\}$. The elementary inequality $\ln u\le u-1$, with equality only at $u=1$, gives, for every $x\in S$,ConsequentlyThis argument applies to a countable discrete probability distribution too: the difference between the two sides of the pointwise inequality is nonnegative, and the negative part of the entropy summand is bounded by $cq(x)$, hence summable. Thus no subtraction of divergent series is involved. Equality forces both ${\sum }_{S}q=1$ and equality in every pointwise inequality; hence $q(x)=p(x)$ on $S$ and $q=0=p$ off $S$. Conversely $p=q$ plainly gives zero. Therefore $\boxed{D(p\Vert q)\ge 0,D(p\Vert q)=0⟺p=q}$.

For the log-sum inequality, set $F={\sum }_{A}f$, $G={\sum }_{A}g$, and normalize to the probability distributions $p_A=f/F$, $q_A=g/G$. When $A$ is nonempty, positivity ensures $F,G>0$, and direct expansion givesThus normalization reduces the log-sum inequality to Gibbs inequality. Equality holds exactly when $f/F=g/G$. The empty-set case is the zero identity under the usual zero-mass convention.

For the final bound, partition the alphabet using $A=\{x:p(x)>q(x)\}$ and put $a=p(A)$, $d=q(A)$. Since the total signed difference is zero,Apply the log-sum inequality separately to $A$ and its complement. The same normalization argument works for countable sets of finite total mass, using the countable Gibbs inequality just proved. It yields the binary partition bound for relative entropyZero block masses are interpreted by the stated conventions; a positive mass divided by zero makes the bound infinite. The given binary estimate now impliesThis is Pinsker's inequality. Equivalently, for the total variation distance $\Vert p-q{\Vert }_{\mathrm{TV}}=\frac{1}{2}{\sum }_x|p(x)-q(x)|$, the bound is $D(p\Vert q)\ge 2{\log }_{b}e\Vert p-q{\Vert }_{\mathrm{TV}}^2$. The factor ${\log }_{b}e$ keeps the statement valid for either bits or natural-logarithm units.