 Asymptotic equipartition property

For a stationary source, typical long blocks have probability approximately $2^{-nH}$; equivalently normalized self-information converges to the entropy rate.

The Asymptotic Equipartition Property (AEP) is a fundamental concept in information theory that describes the behavior of large sequences of random variables. It essentially states that for a sufficiently large number of independent and identically distributed (i.i.d.) random variables, the joint distribution of those variables becomes concentrated around a typical set of outcomes, which have roughly the same probability. Formally, if \(X_1, X_2, \ldots, X_n\) are i.