 Entropy bounds for a type class

For an attainable empirical distribution $q$ on a fixed finite alphabet of size $m$,Thus $\ln |T_n(q)|$ differs from $n{H}_e(q)$ by at most $m\ln (n+1)$, uniformly over attainable profiles. Information entropy is the leading logarithmic count; the polynomial factor matters at short lengths.^[ref]

The upper bound has a short explanation. Under an independent and identically distributed source whose law is $q$ itself, each string in $T_n(q)$ has probability $e^{-n{H}_e(q)}$. Multiplying by the number of such strings cannot give a probability above one. A corresponding polynomial lower bound follows from the multinomial coefficient; the Stirling's approximation gives sharper prefactors when needed.

For the balanced four-symbol class, $H_e(q)=\ln 2$, so the upper bound is $e^{4\ln 2}=16$, while the exact size is six. Information entropy is not an exact count for a short string. On the leading scale it becomes precise: the difference divided by $n$ tends to zero as $n$ grows with the alphabet fixed.