Past exam of the information theory course of the University of Oxford / 2023 (A16551W1, B8.4)

Source paper: www.whatdotheyknow.com/request/ba_computer_science_past_papers/response/2577745/attach/2/202402%20183%20CS%20Papers.zip?cookie_passthrough=1

Solutions to all three questions.^[ref] All logarithms are base two; information entropy is measured in bits and code rates in bits per channel use. Terms $0\log 0$ are zero. Question 3(b) contains an incorrect hint; its requested conclusion is proved directly below for deterministic channel codes. The repeat-until-success protocol in Question 3(c) requires $q<1$ to terminate.

The inequality is true for ordinary addition, and also for addition in a group. For any fixed $y$, the translation $x\mapsto x+y$ is injective, so it merely relabels the probability masses in the conditional distribution. Since $X,Y$ are independent random variables,Averaging over $y$ and using conditioning reduces entropy proves entropy monotonicity under independent addition:There is no requirement that $X+Y$ remain in the original finite alphabet. The proof also gives $H(X+Y)\ge H(Y)$ by symmetry. Independence matters: for a nonconstant $X$ and $Y=-X$, the sum is constant while $H(X)>0$.

Let $N=|\mathcal{X}|\ge 2$ and let $E=1_{\{X\ne Y\}}$ be the error indicator random variable. Because $E$ is determined by $(X,Y)$, the chain rule for conditional entropy in its two orders givesBy conditioning reduces entropy, $H(E\mid Y)\le H(E)$. On $E=0$, knowing $Y$ determines $X$, so the corresponding conditional entropy is zero. On $E=1$, at most $N-1$ symbols remain possible for $X$. The maximum Shannon entropy of a probability distribution on at most $N-1$ symbols is $\log (N-1)$. ConsequentlyFor an arbitrary function $g:\mathcal{X}\to \mathcal{X}$, repeat the argument with $E_g=1_{\{X\ne g(Y)\}}$. This is still determined by $(X,Y)$; when it is zero, $X=g(Y)$, and when it is one, $X$ has at most $N-1$ possibilities for each $Y$. Thus Fano's inequality givesEvents of zero probability contribute zero to the conditional average, covering the error probabilities zero and one. If $N=1$, the left side and error probabilities are zero; the trivial conclusion is handled separately instead of writing $0\log 0$ as an ordinary finite product.

For the entropy summand $f(x)=-x\log x$, differentiation on $(0,1)$ givesSince $f(x)\to 0$ as $x\downarrow 0$, its continuous extension to $[0,1]$ is a concave function.

Fix $0<v<1$ and define the increment $D_v(t)=f(t+v)-f(t)$ on $[0,1-v]$. In its interior,Thus the increment is decreasing, and its endpoint values are $D_v(0)=f(v)$ and $D_v(1-v)=-f(1-v)$. ThereforeThe cases $v=0,1$ follow directly from the endpoint values.

To obtain the sharper half-interval bound, put $G(v)=f(v)-f(1-v)$. It has $G(0)=G(1/2)=0$, whileHence $G$ is a concave function on $[0,1/2]$ and lies above the chord joining its zero endpoints. Thus $f(v)\ge f(1-v)$ there, proving

Set $N=|\mathcal{X}|$, $d_i=|p_i-q_i|$ and $d={\sum }_i{d}_i=\Vert p-q{\Vert }_1$. If $d=0$, the two probability mass functions agree and the bound is zero. Otherwise every $d_i\le d\le 1/2$. The half-interval entropy summand bound, with $t=\min (p_i,q_i)$ and $v=d_i$, givesBy the triangle inequality, followed by Jensen's inequality for the concave function $f$,This proves the requested finite-alphabet entropy continuity bound:

The independent and identically distributed hypothesis factors the joint probability mass function:Symbols outside the support of a probability distribution $p$ never occur, almost surely. On its finite positive probability support, $\log p(X_i)$ is bounded, so the strong law of large numbers applies:This is the asymptotic equipartition property with the sign appropriate to a logarithm of a probability mass.

For a fixed estimated probability mass function $q$, the estimated joint probability mass function is $q_n(x_1,\dots ,x_n)={\prod }_{i}q(x_i)$. If $q(x)>0$ whenever $p(x)>0$, the strong law of large numbers gives the mismatched asymptotic equipartition property:Here $H(p,q)$ is cross-entropy, and the last equality is cross-entropy equals entropy plus relative entropy. The Kullback-Leibler divergence measures the extra average logarithmic loss from the estimated law.

If $q(x)=0$ for some $x$ with $p(x)>0$, let $a={\sum }_{x:q(x)=0}p(x)>0$. The probability that none of the first $n$ observations belongs to this set is $(1-a{)}^n\to 0$. Almost surely such a symbol eventually occurs, after which $q_n(X_1,\dots ,X_n)=0$ forever. Thus the limit is $-\infty$, consistent with infinite cross-entropy and infinite Kullback-Leibler divergence.

For every $x^n\in A^{n,\epsilon }$, the defining typical set constraint impliesLet $S_n=A^{n,\epsilon }\cap B^{n,\epsilon }$. Summing the lower probability mass bound over $S_n$ givesThereforeThe sample mean constraint can only shrink the typical set, so this cardinality upper bound holds for every positive integer $n$.

For fixed $\epsilon >0$, the asymptotic equipartition property gives $\mathrm{Pr}(X^n\notin A^{n,\epsilon })\to 0$. The strong law of large numbers applied to $X_i$, which are bounded because their numerical alphabet is finite, gives $\mathrm{Pr}(X^n\notin B^{n,\epsilon })\to 0$. The union bound therefore yieldsSince $\delta >0$, eventually $\mathrm{Pr}(X^n\in S_n)\ge 2^{-\delta }$. Each word in $S_n$ has probability mass at most $2^{-n(H(p)-\epsilon )}$, soRearranging provesfor sufficiently large $n$. The key is that the factor $2^{-\delta }$ is a fixed number below one, however small $\delta$ is. This is an instance of typical sets with an additional high-probability constraint.

First, the binary translation of a quaternary prefix code is still a prefix code: every translated codeword length is even, and a binary prefix relation would split into complete two-bit blocks, giving a prefix relation between the original quaternary codewords. Binary Huffman coding is optimal among binary prefix codes, soFor the other direction, let ${\ell }_i=|c_B(x_i)|$ and round these binary codeword lengths to quaternary lengths $k_i=\lceil {\ell }_i/2\rceil$. The Kraft inequality givesThe quaternary converse of the Kraft inequality supplies a quaternary prefix code with lengths $k_i$. Optimality of the quaternary Huffman code then impliesThus binary translation of an optimal quaternary code costs at most one extra bit per source symbol:The quaternary Huffman code uses the usual adjustment of adding zero-weight dummy leaves when needed. If the alphabet has one symbol, both optimal codes may use the empty codeword, giving the trivial endpoint.

The specified transition law is a binary erasure channel. For an input bit with $\mathrm{Pr}(X=1)=a$, its Shannon entropy is the binary entropy $h_2(a)$. A non-erased output determines $X$, whereas the erasure symbol leaves its probability distribution unchanged because the erasure event is independent of $X$. ThusMaximizing the mutual information over the input probability distribution, $h_2(a)\le 1$ with equality at $a=1/2$, givesThis includes $q=0$, a noiseless binary channel, and $q=1$, a completely erased channel with zero channel capacity.

Use the standard deterministic channel encoder and channel decoder, with $m$ message labels and code rate $R=(\log m)/n$. The Fano's inequality hint printed here replaces $\log (m-1)$ by $\log m-1$, which is false in general: take $X$ uniform on $\{0,1\}$ and $Y=2$ in the common alphabet $\{0,1,2\}$. The error probability is one and its binary entropy is zero, so the proposed bound asserts $1\le \log 3-1<1$. The correct Fano's inequality alone gives $(1-P_e)\log m\le n(1-q)+h_2(P_e)$, which does not immediately give the requested additive $1/n$. The following erasure-pattern counting bound for maximal decoding error proves the displayed claim directly.

The case $m\le 2$ is immediate because $R\le 1/n$. Suppose $m>2$, put $L=\log m-1>0$, and let $w_0$ be the message that the channel decoder returns on the all-erased output. Choose $W$ uniformly among the other $m-1$ messages. Every one of these messages is decoded correctly with probability at least $1-\delta$, by the assumed maximal decoding error probability bound.

Let $S$ be the random set of unerased coordinates and $K=|S|$. For each fixed $S$ with $K=k\ge 1$, at most $2^k$ output words are possible. A deterministic channel decoder can correctly identify at most one message per output word, so its conditional success probability, averaged over these $m-1$ messages, is at most $\min \{1,2^k/(m-1)\}$. When $K=0$, success is zero because $w_0$ was excluded.

For every integer $k\ge 1$,If $k\ge L$, this is immediate. If $k<L$, write $r=m/2^k>2$. The elementary inequality $\log r\le r-1$ for $r\ge 2$ giveswhere the middle inequality is $(k-1)(r-2)\ge 0$. This proves the needed scalar bound.

Taking the expectation over erasure patterns, and using $\mathbb{E}K=n(1-q)$, yieldsConsequently the requested bound is valid:The use of maximal decoding error probability matters: excluding one message preserves the lower bound on success. The deterministic channel decoder convention also matters; allowing independent randomized guessing would invalidate this particular finite-length claim at $q=1$.

For $0\le q<1$, use repeat-until-success coding over an erasure channel: delete every erasure symbol from the received sequence and retain the non-erased bits in order. Each message bit contributes a string of erasures followed by exactly one correct non-erased copy. The channel encoder advances to the next bit only after this successful copy, so the retained bits are precisely $W_1,\dots ,W_n$. HenceIndeed, $\mathrm{Pr}(M_j>k)=q^k\to 0$, so each stage and the finite collection of $n$ stages terminate almost surely. The protocol uses channel feedback: Alice must know whether an erasure occurred before deciding whether to repeat. This feedback is implicit in the stated algorithm and is not counted as a forward channel use.

At $q=1$, no stage ever finishes, so there is no terminating zero-error decoder for arbitrary messages. This endpoint is excluded from the operational claim above.

For $0\le q<1$, the number of attempts has the geometric distribution with success probability $1-q$:The tail-sum formula for expectation givesAt $q=0$, $M_1=1$ deterministically. At $q=1$, $M_1=+\infty$ and its expectation is $+\infty$, rather than a finite waiting time.

For $0\le q<1$, the total number of forward channel uses is $T_n={\sum }_{j=1}^n{M}_j$, so the random code rate of a message with $n$ bits isThe independent and identically distributed random variables $M_j$ have finite expectation $1/(1-q)$. By the strong law of large numbers,Thus the protocol reaches the binary erasure channel channel capacity in its asymptotic forward throughput while guaranteeing zero error on every completed message. Its random transmission length and use of channel feedback distinguish it from the fixed-length channel codes in part (b). Independent erasure indicators suffice for this calculation; the message bits themselves need not be independent or uniformly distributed. At $q=1$, the algorithm never produces a completed transmission, so $R_n$ as a completed-code rate is undefined; the physical throughput is zero.