Mismatched asymptotic equipartition property

ID: mismatched-asymptotic-equipartition-property

For independent and identically distributed random variables with probability mass function on a finite alphabet, evaluate their realized strings under a fixed product model . If is positive on the probability support of , the strong law of large numbers gives
almost surely, using cross-entropy equals entropy plus relative entropy. If vanishes at an outcome of positive probability, that outcome eventually appears almost surely, making all subsequent product likelihoods zero and the limit . This matches infinite cross-entropy.

New to topics? Read the docs here!