Writing $Q_j$ for the $j$th marginal and using $P=\prod_jP_j$,
$$
D(Q\Vert P)=-H(Q)+\sum_j\mathbb E_{Q_j}[-\log P_j(Y_j)].
$$
The analogous formula for $D(Q^{(i)}\Vert P^{(i)})$ omits coordinate $i$. Consequently
$$
\begin{aligned}
\sum_i\{D(Q\Vert P)-D(Q^{(i)}\Vert P^{(i)})\}
&=-nH(Q)+\sum_iH(Q^{(i)})\\
&\quad+\sum_j\mathbb E_{Q_j}[-\log P_j(Y_j)].
\end{aligned}
$$
Part b makes $\sum_iH(Q^{(i)})\geq(n-1)H(Q)$, so the last display is at least $D(Q\Vert P)$.


Solution

Past exam of the mathematics course of the University of Cambridge / 2021 / iii / Paper 224 / 2 / c / Solution

 Solution

ID: past-exam-of-the-mathematics-course-of-the-university-of-cambridge/2021/iii/paper-224/2/c/solution