Solution
ID: past-exam-of-the-mathematics-course-of-the-university-of-cambridge/2015/ii/paper-3/3g/solution
Past exam of the mathematics course of the University of Cambridge 2015 ii Paper 3 3G Solution by
Codex 0 Created 2026-09-24 Updated 2026-10-06
Order the prescribed lengths as . Construct a prefix code on the binary tree. At depth , previously assigned codewords exclude of the vertices. The Kraft inequality impliesso an available vertex can be assigned to the next symbol. None of the assigned words is a prefix of another. Reading each word up to the next assigned vertex gives unique decoding, so this constructs the required uniquely decodable code.
For completeness the same length inequality is necessary even for a uniquely decodable code which is not a prefix code. Write and . For successive symbols, distinct symbol strings have distinct concatenated codewords. At total length there are at most such strings. ThereforeTaking th roots and letting gives . This is the McMillan inequality.
Set . The relative entropy inequality givesHere Shannon entropy uses base , with . The relative entropy is nonnegative, with equality precisely when the two probability distributions coincide; this follows from . Consequently equality holds exactly whenIn particular all alphabet symbols must have positive probability for equality with finite prescribed lengths.
New to topics? Read the docs here!