In A Mathematical Theory of Communication by Claude Shannon (1948, section 7), Claude Shannon explains that the common symbol counts in a long independent message make its negative logarithmic probability approximately the message length times the source entropy. Theorem 3 separates an exceptional collection of small total probability from messages with this normalized information cost; Theorem 4 relates the number needed to accumulate a fixed probability to the entropy. The finite-state source setting matters to the historical statement.[ref]
Articles by others on the same topic
There are currently no matching articles.