Past exam of the mathematics course of the University of Cambridge / 2024 / iii / Paper 224 / 4 / c / Solution

Part b shows that $H(Q_{\alpha })$ decreases continuously from $H(Q_0)={\log }_2|\mathcal{A}|$ to $H(Q_1)=H(Q)$. Because $Q$ is nonuniform, the variance in part b is positive, so for each intermediate $R$ there is a unique ${\alpha }^{\ast }\in (0,1)$ with $H(Q_{{\alpha }^{\ast }})=R$.

To minimize $D(P\Vert Q)$ subject to $H(P)\ge R$, the optimum lies on the boundary $H(P)=R$. The Lagrange multiplier equations forgive $P(x)\propto Q(x{)}^{\alpha }$ for some $\alpha \in (0,1)$. The entropy constraint selects $\alpha ={\alpha }^{\ast }$ uniquely. Therefore