Past exam of the mathematics course of the University of Cambridge / 2024 / iii / Paper 224 / 2

Let $S(\alpha )={\sum }_{y}Q(y{)}^{\alpha }$. Both $\log S(\alpha )$ and $1-\alpha$ vanish at one. By L'Hopital rule, with logarithms to base two,Terms with $Q(y)=0$ contribute zero by continuity.

The code-distribution correspondence starts from the Kraft inequality. For codeword lengths $L(x)$ putConversely, a probability mass function determines ideal lengths $-{\log }_{2}R(x)$, up to integer rounding. For the source law $P_n$,using nonnegativity of Kullback-Leibler divergence and $K\le 1$.

Use the distribution $R$ associated with the code in part b. Since $2^{L(x)}=(KR(x){)}^{-1}$,For $\alpha =1/(1+\rho )$, Holder inequality, equivalently the indicated Jensen inequality, givesTherefore

Let $\rho \downarrow 0$, so $\alpha =1/(1+\rho )\to 1$. For the finite code alphabet,by differentiating the logarithmic moment-generating function at zero. Part a gives $H_{\alpha }(X_1^n)\to H(X_1^n)$, so the inequality in part c converges to $\mathbb{E}L\ge H(X_1^n)$.