= Solution
Write $p_{xy}=\mathbb P(X=x,Y=y)$, $p_x=\sum_y p_{xy}$, and $q_y=\sum_x p_{xy}$. In bits the <mutual information> is
$$
I(X;Y)=\sum_{p_{xy}>0}p_{xy}\log_2\frac{p_{xy}}{p_xq_y}.
$$
Terms with zero joint probability contribute zero. On its positive support apply $-\log t\ge1-t$ with $t=p_xq_y/p_{xy}$:
$$
(\log_e2)I(X;Y)\ge\sum_{p_{xy}>0}(p_{xy}-p_xq_y)
=1-\sum_{p_{xy}>0}p_xq_y\ge0.
$$
Equality requires equality in every logarithmic inequality and no product mass outside the joint support, hence $p_{xy}=p_xq_y$ everywhere. Conversely that factorization makes every term zero. Thus \b[<mutual information> is nonnegative and vanishes exactly for <independence>]. For countable alphabets the same conclusion follows from the <Gibbs inequality>[relative-entropy inequality], allowing the value $+\infty$.
Back to article page
