Past exam of the mathematics course of the University of Cambridge / 2026 / iii / Paper 224 / 4

For $\mu >0$, let $Q$ be the geometric distributionIf $P$ is the law of $X$, Gibbs inequality givesThereforeFor $\mu =0$, the nonnegative random variable $X$ is zero almost surely and both sides vanish. This proves that the maximum entropy distribution on the nonnegative integers with fixed expected value is geometric.

Solved by gpt-5.6-sol high.

Order the symbols so that $p_1\ge p_2\ge \cdots$. For each $l\ge 0$, there are exactly $2^l$ binary strings of length $l$, while precisely the indicessatisfy $\lfloor \log i\rfloor =l$. Assign those $2^l$ symbols bijectively to the strings of length $l$. The resulting map is an injective function and hence a one-to-one source code, with $L^{\ast }(x_i)=\lfloor \log i\rfloor$. Assigning shorter available words to more probable symbols also shows that this is an optimal one-to-one binary code.

Solved by gpt-5.6-sol high.

Monotonicity of the ordered probabilities gives $i{p}_i\le {\sum }_{j\le i}{p}_j\le 1$, so $\log i\le -\log p_i$. ConsequentlyGiven $L^{\ast }(X)=l$, the source symbol can take at most $2^l$ values. The maximum entropy distribution on a finite set is uniform, henceAveraging this conditional entropy inequality proves

Solved by gpt-5.6-sol high.

Put $L=L^{\ast }(X)$ and $\mu =\mathbb{E}L$. Since $L$ is a function of $X$, the chain rule for information entropy and part (c) givePart (a), applied to the nonnegative integer-valued random variable $L$, yieldsThe logarithm inequality $\log (1+t)\le t\log e$ implies $\mu \log (1+1/\mu )\le \log e$. ThusPart (c) also gives $\mu \le H(X)$, so monotonicity of the logarithm lets us replace $\log (1+\mu )$ by $\log (H(X)+1)$. Rearranging proves

Solved by gpt-5.6-sol high.