= Shannon's typical-message formulation
{c}
In <A Mathematical Theory of Communication by Claude Shannon> (1948, section 7), <Claude Shannon> explains that the common symbol counts in a long independent message make its negative logarithmic probability approximately the message length times the source entropy. Theorem 3 separates an exceptional collection of small total probability from messages with this normalized information cost; Theorem 4 relates the number needed to accumulate a fixed probability to the entropy. The finite-state source setting matters to the historical statement.<A Mathematical Theory of Communication by Claude Shannon>[section 7, Theorems 3–4, reprint pp. 13–14; Appendix 3]{ref}
Back to article page
