Past exam of the mathematics course of the University of Cambridge / 2021 / iii / Paper 224 / 2

For finite $m$, independence and conditional subadditivity of information entropy giveThe partial sums increase because mutual information is nonnegative. Taking $m\to \infty$ proves $H(Z)\ge {\sum }_{i\ge 1}I(X_i;Z)$.

For $S_i=[n]\setminus \{i\}$, the chain rule for information entropy givesEach summand is at least $H(X_j\mid X_1^{j-1})$ because conditioning reduces entropy. Summing over $i$, each $j$ occurs $n-1$ times:This is the required special case of Shearer's inequality.

Writing $Q_j$ for the $j$th marginal and using $P={\prod }_j{P}_j$,The analogous formula for $D(Q^{(i)}\Vert P^{(i)})$ omits coordinate $i$. ConsequentlyPart b makes ${\sum }_{i}H(Q^{(i)})\ge (n-1)H(Q)$, so the last display is at least $D(Q\Vert P)$.