= Bernoulli typical-set example
{title2=$p=1/4,\quad H=h_2(1/4)\approx0.811278$}
Take an <IID random variable> source of <bits> with $P(X_j=1)=1/4$. Its <binary entropy> is
$$
H=h_2(1/4)=-\frac14\log_2\frac14-\frac34\log_2\frac34
\approx0.811278\text{ bits per symbol}.
$$
A word of length $n$ containing $k$ ones has <probability> $(1/4)^k(3/4)^{n-k}$. Its <information content> per symbol is
$$
\frac{\imath(k)}n=\log_2\frac43+\frac kn\log_2 3.
$$
Thus <weakly typical sequences> for this biased source have $k/n$ near $1/4$. With tolerance $\varepsilon=0.1$ <bits> per symbol, the exact condition is
$$
\left|\frac kn-\frac14\right|\leq\frac{0.1}{\log_2 3}.
$$
For $n=40$, this includes exactly $k=8,9,10,11,12$. There are $\binom{40}{k}$ words with each count: a <type class> with that many possible orderings. Summing the counts and their <probability masses> gives
$$
\boxed{|T_{0.1}^{(40)}|=9\,096\,659\,013,\qquad
P(T_{0.1}^{(40)})\approx0.638912.}
$$
These are fewer than one percent of the $2^{40}$ possible words, yet they carry nearly 64 percent of the <probability>. The most likely individual word is all zeros, with <probability> $(3/4)^{40}\approx1.006\times10^{-5}$; it is outside this <typical set>. No contradiction is involved: the <type classes> near ten ones each contain many more individual words.
\Image[/typical-set-individual-words-and-count-mass.png]
{title=The most likely individual word differs from the counts carrying most probability}
{description=The left plot gives the <information content> of one word with each number of ones. The right includes the <binomial coefficient> counting every such word, giving the exact <binomial distribution> of the count. Both use $n=40$, $p=1/4$; the green band marks the <typical set> with tolerance $0.1$ <bits> per symbol.}
Back to article page
