Past exam of the communication Theory course of the University of Oxford / 2009 / 1 / a / ii / Solution

Apply Gibbs inequality to the joint distribution and the product of its marginals. Restrict to the Cartesian product of their positive supports, so the reference probabilities are strictly positive as the supplied inequality requires. Expanding givesThis proves the second inequality. For the first, apply the same argument to the conditional joint distribution of $(X,Z)$ for each $y$ with $p_Y(y)>0$, and average with those probabilities:Here the conditional support restriction again removes zero marginal probabilities. ThusThe nonnegative differences are conditional mutual information and mutual information, respectively.