Solution

ID: past-exam-of-the-communication-theory-course-of-the-university-of-oxford/2010/1/c/solution

For the nontrivial case , every and
The probability ordering makes . At a dyadic number, choose the terminating binary expansion followed by zeros. Let be the first bits of , and let
be its binary cylinder interval.
Suppose and were a prefix of . Since , the numbers and would have the same first bits, so both would lie in . In particular . But
a contradiction. A later word cannot be a shorter prefix of an earlier word because the lengths are nondecreasing; equal-length coincidences were also excluded. Thus the cumulative Shannon code is a prefix code.
The ceiling inequalities give
Multiply by and sum to obtain
For a one-symbol deterministic source the displayed construction assigns the empty word and gives . This zero-bit convention is valid when the number of source emissions is supplied separately. It is not an unframed uniquely decodable code: concatenating different numbers of empty words gives the same bit string. If positive-length unframed words are required, the shortest deterministic code has length , so the strict upper bound requires the nontrivial-source assumption.

New to topics? Read the docs here!