 Solution

Use base-two logarithms and the convention $0{\log }_{2}0=0$. Write $p_y=\mathbb{P}(Y=y)$ and $q_y(x)=\mathbb{P}(X=x\mid Y=y)$ for $p_y>0$. The conditional entropy is the average of the information entropies of these conditional laws:The careful form of Gibbs inequality is that, for two probability mass functions $q,p$ on a common countable set,where a term with $q(x)>0=p(x)$ gives $+\infty$. Equality holds exactly when $q=p$. This is the nonnegativity of relative entropy.

First suppose $H(X)<\infty$. For every $y$ of positive probability, $q_y(x)>0$ implies $p_X(x)>0$. Averaging the cross-entropies givesAll the summands here are nonnegative, so interchanging the sums is valid, and each conditional cross-entropy is finite. Applying Gibbs inequality to each $q_y,p_X$ shows that $H(q_y)$ is at most its cross-entropy. Subtracting and averaging therefore givesEvery term on the right is nonnegative. The equality in the entropy conditioning inequality is thus equivalent to $q_y=p_X$ for all $p_y>0$, which is exactly factorization of the joint probability mass function. Consequently

For arbitrary countable discrete random variables, the inequality remains valid in $[0,\infty ]$: the case $H(X)=\infty$ is immediate. The finite-entropy condition is necessary for the usual equality characterization. For example, let $\mathbb{P}(A=n)=c/(n(\ln n{)}^2)$ for $n\ge 2$, with $c$ the normalizing constant, and let $B$ be an independent fair bit. The series defining the probability mass function converges, whereas its information entropy diverges like ${\sum }_{n}1/(n\ln n)$. Then $X=(A,B)$ and $Y=B$ are dependent, but $H(X)=H(X\mid Y)=\infty$. In the infinite case, equality means precisely that the conditional entropy is also infinite; it does not imply independent random variables.