= Entropy rate
{title2=$h(X)=\lim_{n\to\infty}H(X_1,\ldots,X_n)/n$}
{wiki}
= Information rate
{synonym}
The <entropy rate> measures average new uncertainty per emitted symbol of a <stochastic process>. Predictability from earlier symbols can make this much smaller than the <information entropy> of one symbol. For example, a <binary Markov source> can emit a fair <bit> at every time while repeating its previous <bit> with <probability> $7/8$: each individual symbol has one <bit> of uncertainty, but the <entropy rate> is only about $0.544$ <bits> per symbol.
For a <finite alphabet> source $X=(X_1,X_2,\ldots)$, define
$$
h(X)=\lim_{n\to\infty}\frac{H(X_1,\ldots,X_n)}n
$$
when the <limit> exists. Base-two <logarithms> give <bits> per symbol. Multiplying by a fixed emission frequency $f$ gives $fh(X)$ <bits> per unit time; the symbol rate and the uncertainty of the sequence are different inputs. A <strictly stationary process> on a <finite alphabet> always has an <entropy rate>, equal to its <conditional entropy rate>.<6.441 Information Theory (Spring 2016 lecture notes) by Yury Polyanskiy and others>[§5.5, Definition 5.1 and Theorem 5.5, printed p. 57: block entropy and stationary conditional-entropy limit]{ref}
For an <independent and identically distributed> source, the <chain rule for information entropy> gives $H(X_1,\ldots,X_n)=nH(X_1)$, hence $h(X)=H(X_1)$. With dependence, <conditioning reduces entropy> and makes the past useful. The examples under <Markov information source> show this benefit using exact finite-block calculations.
Back to article page
