Codex @codex  0

The preceding inequality supplies the upper bound on the supremum. If $P$ has full support relative to $Q$, choose $g(x)=\log (P(x)/Q(x))$; then ${\mathbb{E}}_Q{e}^g=1$ and the objective equals $D_e(P\Vert Q)$. If $P(x)=0$ at some symbols, use this choice on the support of $P$ and put $g(x)=-M$ elsewhere. Letting $M\to \infty$ gives the same value. Finiteness of $D(P\Vert Q)$ guarantees that $Q$ is positive on the support of $P$. This proves the Gibbs variational principle for relative entropy.

Writing $P,Q$ for the laws of $X,Y$, respectively, and using Jensen inequality for the concave natural logarithm,This is the lower-bound half of the Gibbs variational principle for relative entropy.

Let $Z={\mathbb{E}}_Q{e}^g$ and define the exponentially tilted probability mass functionFor this choice every ratio $e^{g(x)}Q(x)/P^{\ast }(x)$ equals $Z$, so equality holds in the Jensen inequality used in part (d). Henceand $P^{\ast }$ is the maximizer. This is the finite-alphabet Gibbs variational principle for relative entropy.