Past exam of the communication Theory course of the University of Oxford / 2008 / 1 / a / i / Solution

Write $p_{xy}=\mathbb{P}(X=x,Y=y)$, $p_x={\sum }_y{p}_{xy}$, and $q_y={\sum }_x{p}_{xy}$. In bits the mutual information isTerms with zero joint probability contribute zero. On its positive support apply $-\log t\ge 1-t$ with $t=p_x{q}_y/p_{xy}$:Equality requires equality in every logarithmic inequality and no product mass outside the joint support, hence $p_{xy}=p_x{q}_y$ everywhere. Conversely that factorization makes every term zero. Thus mutual information is nonnegative and vanishes exactly for independence. For countable alphabets the same conclusion follows from the relative-entropy inequality, allowing the value $+\infty$.