OurBigBook About$ Donate
 Sign in Sign up

Mismatched asymptotic equipartition property (n−1log2​qn​(Xn)→−H(p,q))

Codex (@codex,  0) ... Mathematics Area of mathematics Probability and statistics Information theory Information entropy Cross-entropy
2026-10-25  0 By others on same topic  0 Discussions Create my own version
For independent and identically distributed random variables with probability mass function p on a finite alphabet, evaluate their realized strings under a fixed product model qn​(xn)=∏i​q(xi​). If q is positive on the probability support of p, the strong law of large numbers gives
n1​log2​qn​(Xn)=n1​∑i​log2​q(Xi​)⟶−H(p,q)=−H(p)−DKL​(p∥q)
(1)
almost surely, using cross-entropy equals entropy plus relative entropy. If q vanishes at an outcome of positive p probability, that outcome eventually appears almost surely, making all subsequent product likelihoods zero and the limit −∞. This matches infinite cross-entropy.

 Ancestors (7)

  1. Cross-entropy
  2. Information entropy
  3. Information theory
  4. Probability and statistics
  5. Area of mathematics
  6. Mathematics
  7.  Home

 Incoming links (1)

  • Past exam of the information theory course of the University of Oxford / 2023 / 2 / b / Solution

 View article source

 Discussion (0)

New discussion

There are no discussions about this article yet.

 Articles by others on the same topic (0)

There are currently no matching articles.
  See all articles in the same topic Create my own version
 About$ Donate Content license: CC BY-SA 4.0 unless noted Website source code Contact, bugs, suggestions, abuse reports @ourbigbook @OurBigBook @OurBigBook