Optimal prefix code (${\min }_{c}L(c)$)

An optimal prefix code minimizes expected codeword length for a fixed source probability distribution. Here the source alphabet is finite, has $n\ge 2$ symbols, and has $p_x>0$ for every symbol; each source symbol receives one codeword over the binary output alphabet $\{0,1\}$. Its code tree puts the symbols at leaves. Reading the edges along the graph path from the root of a tree to a leaf produces its codeword, so the leaf depth in a code tree $d_x$ is exactly its codeword length. ThusFor unnormalized positive weights, minimize $C(T)={\sum }_x{w}_x{d}_x$ instead; the corresponding expected codeword length is $C(T)/{\sum }_x{w}_x$. This connects the abstract optimization to storage or transmission: $d_x$ counts the bits charged when symbol $x$ occurs. Huffman coding attains this minimum; the Huffman sibling property and Huffman sibling contraction explain why. ^[ref]

For $(p_A,p_B,p_C,p_D)=(0.4,0.3,0.2,0.1)$, a balanced code tree charges every symbol two bits, giving $L=2$. The codeword lengths $(1,2,3,3)$ charge the most frequent symbol only one bit, giving $L=1.9$. The choice concerns weighted cost, not just the greatest codeword length: increasing that maximum from two to three can improve the average.

The search over prefix codes looks infinite, but positivity reduces it to finitely many full binary trees with $n$ leaves. A vertex with one child can be suppressed to improve cost, and a full binary tree with $n$ leaves has $n-1$ internal vertices. There are consequently finitely many such shapes and symbol assignments, so the minimum exists. For four leaves, the only possible sorted codeword lengths in a full binary tree are $(2,2,2,2)$ and $(1,2,3,3)$. Probability monotonicity of optimal prefix-code lengths assigns larger probabilities to shallower leaves, giving respective minimum costs $2$ and $1.9$ in this example. This is an independent small-instance check of the Huffman code.

These hypotheses matter. With zero weights, suppressing a subtree carrying only zero weight need not strictly improve cost, so an optimal prefix code need not have a full binary tree. A one-symbol source raises the convention of whether an empty codeword is allowed. Both cases are outside the original statement here. Encoding groups of symbols changes the source alphabet and the optimization problem; the minimum here concerns one source symbol per codeword.

For a strictly positive probability distribution on $2^n$ source symbols, with $n>1$, every optimal binary prefix code has common length $n$ if and only if $p_x+p_y>p_z$ for every three distinct symbols. It suffices to compare the sum of the two smallest probabilities with the largest.

An optimal binary prefix tree has no node with only one child, because removing that detour would strictly shorten positive-probability words. It therefore satisfies equality in the Kraft inequality. If it is not a fixed-length tree, it has a deepest sibling pair at depth $L>n$ and a short leaf at depth $\ell <n$. Exchanging that pair with the short leaf changes the expected codeword length by $(L-\ell -1)(p_z-p_x-p_y)$. Strict positivity of every pair sum over a third probability makes this change negative, contradicting optimality.

Conversely, if $p_x+p_y\le p_z$, begin with the complete depth-$n$ tree, place $x,y$ at sibling leaves, replace their parent by $z$, and replace the previous $z$ leaf by children $x,y$. The new expected length is $n+p_x+p_y-p_z\le n$ and its lengths are unequal. If the optimum is below $n$, every optimal tree is nonuniform; if it is $n$, this modified tree is a nonuniform optimum. Thus equality in the probability condition already defeats the assertion that every optimum has fixed length.

The number of edges on the graph path from the root of a tree to the leaf for symbol $x$ in a code tree. The root of a tree has depth zero. Each edge contributes one bit to a binary codeword, so this depth equals the codeword length; weighting these depths by probabilities gives expected codeword length.

For an equiprobable source, an optimal binary prefix tree has leaf depths differing by at most one. If one leaf has depth $t$ and a deepest sibling pair has depth $s\ge t+2$, split the shallow leaf into two children and collapse the deep pair into its parent. The number of leaves is unchanged and their total depth decreases by $s-t-1>0$, contradicting optimality.

// Target: probability-and-statistics.bigb

For a source symbol $X$ and a code $c$, the expected codeword length is

If symbols of probabilities $p_i>p_j$ have codeword lengths $l_i$ and $l_j$ in an optimal prefix code, then $l_i\le l_j$. Otherwise, exchanging their codewords preserves the prefix property and changes the expected length bycontradicting optimality.

Every optimal binary prefix code has two codewords of maximal length that differ only in their final bit. Indeed, choose a deepest leaf in the prefix tree. If its sibling were not a codeword, maximality of the depth would make that sibling subtree empty, so the chosen word could be shortened by deleting its final bit. This would preserve prefix-freeness and reduce expected length.

An optimal prefix code can be nonunique in several different senses. Exchanging the $0,1$ labels on the outgoing edges of an internal vertex changes codewords while preserving every symbol's codeword length, the code tree shape, and expected codeword length. This freedom exists even with strictly unequal probabilities. Exchanging equal-probability symbols between leaves also preserves expected codeword length; if their leaf depths in a code tree differ, their individual codeword lengths change, though the set of leaf depths does not. For example, three equiprobable symbols have optimal codeword lengths $(1,2,2)$: any of the three can receive the one-bit codeword.^[ref]

Ties between weights of entire subtrees during Huffman coding can go further and change the code tree shape and its collection of codeword lengths. Take $(p_A,p_B,p_C,p_D)=(0.4,0.2,0.2,0.2)$ and first merge $C,D$, creating weight $0.4$. Next, $B$ of weight $0.2$ can merge either with $A$ or with the $CD$ subtree, both of weight $0.4$. The two resulting prefix codes areFor the second prefix code, $L=0.4\cdot 1+0.2\cdot 2+0.2\cdot 3+0.2\cdot 3=2$ bits. Both merge sequences add weights $0.4,0.6,1$, whose sum is $2$. Enumerating the full binary trees with four leaves also proves optimality independently: $(2,2,2,2)$ costs $2$, while the least cost of $(1,2,3,3)$ is again $2$. The two code trees cannot be obtained from each other merely by relabelling leaves or exchanging edge labels: their collections of leaf depths in a code tree differ.

The quantifier in the Huffman sibling property is visible here too. Choose $B,C$ as two least-probability symbols. In the displayed balanced optimal prefix code, they are not sibling leaves in a code tree. Exchanging $B,D$, which have equal probabilities, produces another balanced optimal prefix code in which $B,C$ are deepest sibling leaves in a code tree. Thus the proof supplies an optimal representative with the chosen pair together. It does not force that labelling on every optimum. An implementation can make its tie choices and $0/1$ edge assignments deterministic when encoder and decoder need to reproduce the same codewords; optimal mean length alone does not specify them.