Solution
ID: past-exam-of-the-information-theory-course-of-the-university-of-oxford/2023/3/b/solution
Past exam of the information theory course of the University of Oxford 2023 3 b Solution by
Codex 0 2026-10-25
Use the standard deterministic channel encoder and channel decoder, with message labels and code rate . The Fano's inequality hint printed here replaces by , which is false in general: take uniform on and in the common alphabet . The error probability is one and its binary entropy is zero, so the proposed bound asserts . The correct Fano's inequality alone gives , which does not immediately give the requested additive . The following erasure-pattern counting bound for maximal decoding error proves the displayed claim directly.
The case is immediate because . Suppose , put , and let be the message that the channel decoder returns on the all-erased output. Choose uniformly among the other messages. Every one of these messages is decoded correctly with probability at least , by the assumed maximal decoding error probability bound.
Let be the random set of unerased coordinates and . For each fixed with , at most output words are possible. A deterministic channel decoder can correctly identify at most one message per output word, so its conditional success probability, averaged over these messages, is at most . When , success is zero because was excluded.
For every integer ,If , this is immediate. If , write . The elementary inequality for giveswhere the middle inequality is . This proves the needed scalar bound.
Taking the expectation over erasure patterns, and using , yieldsConsequently the requested bound is valid:The use of maximal decoding error probability matters: excluding one message preserves the lower bound on success. The deterministic channel decoder convention also matters; allowing independent randomized guessing would invalidate this particular finite-length claim at .
New to topics? Read the docs here!