Past exam of the mathematics course of the University of Cambridge / 2025 / iii / Paper 224 / 1

For nonnegative $a_i,b_i$, with $a={\sum }_i{a}_i$ and $b={\sum }_i{b}_i$, the log-sum inequality iswith the usual extended-value conventions. Equality holds when $a_i/b_i$ is constant wherever $a_i>0$.

Let $0\le \lambda \le 1$ and put $P=\lambda P_1+(1-\lambda )P_2$ and $Q=\lambda Q_1+(1-\lambda )Q_2$. For each alphabet symbol $x$, apply the log-sum inequality to $a_1=\lambda P_1(x)$, $a_2=(1-\lambda )P_2(x)$ and the corresponding $b$ values. Summing over $x$ giveswhich is joint convexity in $(P,Q)$.

For $Z={\mathbb{E}}_Q{e}^g$, define the exponentially tilted mass function $Q_g(x)=Q(x)e^{g(x)}/Z$. Gibbs inequality givesRearranging proves

The preceding inequality supplies the upper bound on the supremum. If $P$ has full support relative to $Q$, choose $g(x)=\log (P(x)/Q(x))$; then ${\mathbb{E}}_Q{e}^g=1$ and the objective equals $D_e(P\Vert Q)$. If $P(x)=0$ at some symbols, use this choice on the support of $P$ and put $g(x)=-M$ elsewhere. Letting $M\to \infty$ gives the same value. Finiteness of $D(P\Vert Q)$ guarantees that $Q$ is positive on the support of $P$. This proves the Gibbs variational principle for relative entropy.