Past exam of the information theory course of the University of Oxford / 2021 / 2 / b

The shape of minimum-cost positive-weight ternary prefix trees arguments in ii–vi assume a full-support finite distribution: $p_i>0$ for every retained source symbol. The printed setup does not state this globally. Each affected solution gives the proof with that qualification and a counterexample when zero-probability symbols are allowed in arbitrary redundant code trees. Part i needs no such qualification.

Exchange the two complete codewords while leaving the code tree unchanged. The prefix code property is preserved because the set of codewords is unchanged. If $p_i>p_j$ but $l_i>l_j$, the change in expected codeword length iscontradicting optimality. Hence $p_i>p_j$ implies $l_i\le l_j$. This probability monotonicity of optimal prefix-code lengths is valid even if some probabilities are zero; equal probabilities impose no order.^[ref]

First assume a full-support finite distribution, as stated above. Let $L$ be the largest codeword length and take a tree leaf at depth $L$. Every existing tree child of its tree parent must also be a tree leaf: an internal tree child would have a descendant deeper than $L$. If the chosen tree leaf were its tree parent's only tree child, moving its symbol to that tree parent would shorten its codeword by one, preserve the prefix code property, and reduce expected codeword length by its positive probability. That contradicts optimality. The tree parent therefore has at least two tree children, both tree leaves. A second codeword has the same first $L-1$ digits and a different last digit, proving the required deepest sibling codeword property for a ternary prefix code.

For the literal wording allowing zero probabilities, take $p=(1,0,0)$ and the prefix code $(0,10,110)$. Its expected codeword length is one, which is minimal: with more than one message an empty codeword is impossible, so the positive-probability message must have length at least one. Yet the deepest tree leaf $110$ has no sibling codeword. Thus positivity or an additional convention removing redundant zero-weight branches is needed.

Suppose a tree parent at depth $n-1$, with $n<L$, has an unused outgoing digit. The missing tree child gives an empty code tree branch at depth $n$. Move any deepest tree leaf into that position, retaining all other codewords and deleting the abandoned empty branch. The new word has no old word as a prefix, because it lies in an unused branch, and it is not a prefix of any retained word. Its length decreases from $L$ to $n$. With a full-support finite distribution this reduces expected codeword length by $p_i(L-n)>0$, contradicting optimality. Hence all three tree children exist, and each has a descendant codeword. In fact every internal vertex above depth $L-1$ has all three tree children. Applying this to the tree parent of the specified prefix gives the claimed three extensions.

The positive-probability qualification attached only to the specified prefix does not by itself suffice. In the zero-weight example $(p,c)=((1,0,0),(0,10,110))$, choose the positive-probability word $0$, so $n=1<L=3$. There is no word starting with $2$. The deepest tree leaf available for relocation has probability zero, so the strict-improvement argument fails. This is why the full-support finite distribution assumption is used for the structural conclusions.

With a full-support finite distribution, suppose two distinct tree parents at depth $L-1$ each have exactly two tree children, both tree leaves. Move one tree leaf from the second pair into the vacant third tree child slot of the first tree parent. Its length remains $L$, so this relocation does not change expected codeword length. The second tree parent now has a single tree child, which is a tree leaf. Replace that remaining tree child's codeword by the tree parent's word. This preserves the prefix code property and shortens a positive-probability codeword by one. Expected codeword length strictly decreases, a contradiction. Thus there is at most one deepest pair with exactly two sibling codewords.

If zero-probability symbols are retained without a reduction convention, the claim can fail. The probability mass function $(1,0,0,0,0)$ with prefix code $(0,100,101,110,111)$ has the optimal expected codeword length one, yet contains the two deepest pairs $(100,101)$ and $(110,111)$. The contraction would only shorten a zero-probability word.

Let $L$ again denote the largest codeword length, and assume a full-support finite distribution. The missing-branch exchange from part iii forces every internal vertex at depth at most $L-2$ to have three tree children. At depth $L-1$, all tree children are tree leaves. Part ii rules out one tree child there; the hypothesis that there is no pair of two deepest sibling codewords rules out two tree children. Thus each such tree parent also has three tree children. Every internal vertex has exactly three tree children, which is a full ternary tree, called semi-complete in this paper. The same last-level argument covers $L=1$.

Without the assumption of a full-support finite distribution the implication is false: for $p=(1,0,0)$, the optimal prefix code $(0,10,110)$ has no deepest pair of two sibling codewords, but its code tree contains vertices with a single tree child. Empty unused tree children are not vertices of the code tree.

Assume a full-support finite distribution. The preceding structural arguments show that every internal vertex has three tree children, except possibly one tree parent at depth $L-1$ with two tree children. Let $N_3$ and $N_2$ count these two kinds of internal vertex. Part iv gives $N_2\in \{0,1\}$; no vertex has one tree child.

A finite rooted tree with $m$ tree leaves has $m+N_2+N_3-1$ edges, while counting outgoing edges gives $2N_2+3N_3$. Equating the counts yieldsFor an odd integer $m$, the right side forces $N_2$ to be an even integer. Since it is at most one, $N_2=0$. Thus no deepest pair has only two sibling codewords, and the optimal code tree is a full ternary tree. This leaf-count identity also directly proves that a full ternary tree has an odd number of tree leaves.

Again the unrestricted zero-weight version is false: $(0,10,11)$ is an optimal prefix code for $p=(1,0,0)$, with odd integer $m=3$, but its two deepest tree leaves form a two-sibling pair. There is no contradiction with the proof because its tree root has only two tree children, which the full-support finite distribution assumption would forbid above the deepest level.