= Solution
Use the <continuous> convention $0\log_2(0/q)=0$. Let $S=\{r:p_r>0\}$. Applying the allowed <logarithm> <inequality> to $q_r/p_r$ for $r\in S$ gives
$$
-d(p,q)=\sum_{r\in S}p_r\log_2\frac{q_r}{p_r}
\leq c\sum_{r\in S}(q_r-p_r)=c\left(\sum_{r\in S}q_r-1\right)\leq0.
$$
Hence $d(p,q)\geq0$. <Equality> forces $\sum_{r\in S}q_r=1$. Since every $q_r$ is strictly positive, this makes $S$ the entire <alphabet>. <Equality> in every weighted <logarithm> <inequality> then forces $q_r/p_r=1$ for every $r$. Conversely $p=q$ makes every summand zero. Thus
$$
\boxed{d(p,q)\geq0,\qquad d(p,q)=0\iff p=q.}
$$
This is <Gibbs inequality> for <relative entropy>, with zero <probabilities> in the first argument handled explicitly.
Back to article page
