= Mismatched asymptotic equipartition property
{title2=$n^{-1}\log_2q_n(X^n)\to-H(p,q)$}
For <independent and identically distributed random variables> with <probability mass function> $p$ on a <finite alphabet>, evaluate their realized strings under a fixed product model $q_n(x^n)=\prod_iq(x_i)$. If $q$ is positive on the <probability support> of $p$, the <strong law of large numbers> gives
$$
\frac1n\log_2q_n(X^n)=\frac1n\sum_i\log_2q(X_i)\longrightarrow-H(p,q)=-H(p)-D_{\mathrm{KL}}(p\Vert q)
$$
<almost surely>, using <cross-entropy equals entropy plus relative entropy>. If $q$ vanishes at an outcome of positive $p$ <probability>, that outcome eventually appears <almost surely>, making all subsequent product likelihoods zero and the limit $-\infty$. This matches infinite <cross-entropy>.
Back to article page
