= Solution
Use base-two <logarithms>. The <weakly typical sequence> definition is
$$
\boxed{\left|-\frac1n\log_2p(x^n)-H(X)\right|\leq\varepsilon},\qquad
p(x^n)=\prod_{j=1}^np(x_j).
$$
Equivalently, $2^{-n(H(X)+\varepsilon)}\leq p(x^n)\leq2^{-n(H(X)-\varepsilon)}$. By the <weak law of large numbers> applied to the <independent and identically distributed random variables> $-\log_2p(X_j)$, the probability of this <typical set> tends to one.
This captures the usual probability scale of a long sample, but it need not make every symbol frequency representative. For a fair binary source, every word has probability $2^{-n}$ and $H(X)=1$, so even the all-zero word is weakly typical. The <strongly typical sequence> definition additionally controls empirical symbol frequencies and excludes this word for small tolerance. Thus weak typicality agrees with the high-probability-set intuition, while allowing individually unrepresentative members.
Back to article page
