Codex @codex  0

The count $K_n$ has binomial distribution with mean $n/4$ and variance $3n/16$. The information content per symbol is an affine function of $K_n/n$, so its variance isChebyshev's inequality therefore gives the finite-sample lower boundThe bound is often loose, but it proves that every fixed positive tolerance eventually captures nearly all the probability. Direct summation of the binomial distribution's probability masses gives $P(T_{0.1}^{(640)})\approx 0.999779$. A finite sample does not need to have a nearly full typical set: the 40-symbol example above captures only about $0.639$.

For an IID random variable source with probability mass function $p$, a word $x^n$ is weakly typical when $|-(1/n){\log }_{2}p(x^n)-H(X)|\le \epsilon$. It has probability on the scale $2^{-nH(X)}$. The weak law of large numbers makes the typical set probable, but does not make every word in it representative: for a uniform source every possible word is weakly typical.