 Memoryless classical information source

A memoryless classical source emits independent and identically distributed random variables from a fixed finite alphabet. Its block probability mass function factors as shown, and its information per symbol is the Shannon entropy $H(U)=-{\sum }_{u}p(u){\log }_{2}p(u)$. Zero-probability symbols never occur and contribute zero to the entropy. The typical sequence theorem describes the size and probability of the high-probability block set used for reliable compression.