Past exam of the communication Theory course of the University of Oxford / 2008 / 3

Unique decipherability makes the map from $(X_1,\dots ,X_m)$ to the concatenated binary string $M$ injective. Entropy is unchanged by an injective relabelling, and memorylessness gives independence, henceGiven $L=\ell$, the string has at most $2^{\ell }$ possibilities. The entropy bound for finite alphabets therefore gives $H(M\mid L=\ell )\le \ell$. Averaging,The random length cannot simply be discarded: it can also carry information. Let ${\ell }_{\max }$ be the maximum codeword length, finite since the alphabet is finite. Since $L$ is a function of $M$ and takes at most $m{\ell }_{\max }+1$ values,Divide by $m$ and let $m\to \infty$. This proves $\boxed{\mathbb{E}|c(X)|\ge H(X)}$. The conclusion also covers a deterministic source; the source's lowercase $x$ in its final expectation denotes the random source letter.

The entropy isTo construct a Huffman code, work with integer weights $2,2,3,4,5$. Merge the two twos to make four. Of the remaining weights $3,4,4,5$, merge three with the original four to make seven, and merge the new four with five to make nine. Finally merge seven with nine. Resolving the tie in this way gives, for example,This is a prefix code and Huffman's optimality theorem makes it optimal. Its expected length isDifferent choices at a tied merge can give different optimal codewords with the same expected length.

Let $\bar{\ell }=\mathbb{E}|c(X)|$. The Huffman encoder emits an average of $w\bar{\ell }$ binary digits per minute. Shannon's noisy-channel coding theorem states that for a discrete memoryless channel of capacity $K$, every information rate strictly below $K$ bits per use is achievable with error probability tending to zero as block length grows; rates above $K$ are not achievable for unrestricted messages.

Thus a channel code carrying the Huffman bitstream reliably with buffering and arbitrarily long blocks can be chosen wheneverThe strict inequality supplies positive coding and buffering slack. Huffman's expected-length bound is $H(X)\le \bar{\ell }<H(X)+1$ for a nontrivial finite source. Therefore the simpler bound $r>w(H(X)+1)/K$ is sufficient without knowing its exact Huffman lengths.

Distinguish this bitstream construction from the absolute source-channel limit. The noiseless source coding theorem permits long source blocks to be compressed to any rate above $H(X)$ bits per source symbol, with arbitrarily small overhead. Together with channel coding, this gives the sharper limiting requirement $rK>wH(X)$; $rK<wH(X)$ is impossible by the source-channel converse. Further compression can exploit redundancy remaining in a single-symbol Huffman stream, so $w\bar{\ell }/K$ need not be the universal minimum channel symbol rate. At $K=0$ a positive-entropy source cannot be sent reliably at a finite rate.