Past exam of the communication Theory course of the University of Oxford / 2009 / 1 / a

For finite-valued random vectors, the conditional entropy is the average of the entropies of the conditional distributions:Use $0{\log }_{2}0=0$ and omit zero-probability conditioning events. Equivalently,The base two makes all information quantities measured in bits; changing the logarithm base scales them by a positive constant and preserves the inequalities.

Apply Gibbs inequality to the joint distribution and the product of its marginals. Restrict to the Cartesian product of their positive supports, so the reference probabilities are strictly positive as the supplied inequality requires. Expanding givesThis proves the second inequality. For the first, apply the same argument to the conditional joint distribution of $(X,Z)$ for each $y$ with $p_Y(y)>0$, and average with those probabilities:Here the conditional support restriction again removes zero marginal probabilities. ThusThe nonnegative differences are conditional mutual information and mutual information, respectively.