For a stationary source, typical long blocks have probability approximately ; equivalently normalized self-information converges to the entropy rate.
For an IID source, the epsilon-typical set consists of words satisfying . Its probability tends to one, and its cardinality is at most .
For an IID random variable source with probability mass function , a word is weakly typical when . It has probability on the scale . The weak law of large numbers makes the typical set probable, but does not make every word in it representative: for a uniform source every possible word is weakly typical.
For a typical set of probability at least ,The upper bound sums the lower probability bound for each word; the lower bound sums the upper probability bound and uses the set's total probability. These bounds turn the asymptotic equipartition property into exponential estimates of source-coding dimension.
For a finite-alphabet IID random variable source, a strongly typical word has empirical symbol frequencies within the chosen tolerance of every , and contains no zero-probability symbols. Unlike a weakly typical sequence, this definition controls the individual symbol counts. Both definitions give sets whose probability tends to one for fixed positive tolerance.
Articles by others on the same topic
The Asymptotic Equipartition Property (AEP) is a fundamental concept in information theory that describes the behavior of large sequences of random variables. It essentially states that for a sufficiently large number of independent and identically distributed (i.i.d.) random variables, the joint distribution of those variables becomes concentrated around a typical set of outcomes, which have roughly the same probability. Formally, if \(X_1, X_2, \ldots, X_n\) are i.