= Solution
Part b shows that $H(Q_\alpha)$ decreases continuously from $H(Q_0)=\log_2|\mathcal A|$ to $H(Q_1)=H(Q)$. Because $Q$ is nonuniform, the variance in part b is positive, so for each intermediate $R$ there is a unique $\alpha^*\in(0,1)$ with $H(Q_{\alpha^*})=R$.
To minimize $D(P\|Q)$ subject to $H(P)\geq R$, the optimum lies on the boundary $H(P)=R$. The <Lagrange multiplier> equations for
$$
D(P\|Q)+\lambda\{R-H(P)\}+\nu\left(\sum_xP(x)-1\right)
$$
give $P(x)\propto Q(x)^\alpha$ for some $\alpha\in(0,1)$. The entropy constraint selects $\alpha=\alpha^*$ uniquely. Therefore
$$
D^*(R,Q)=D(Q_{\alpha^*}\|Q).
$$
Back to article page
