= Conditional entropy rate
{title2=$\lim_{n\to\infty}H(X_n\mid X_1,\ldots,X_{n-1})$}
For a <strictly stationary process> on a <finite alphabet>, put $a_1=H(X_1)$ and $a_n=H(X_n\mid X_1,\ldots,X_{n-1})$ for $n\geq2$. Each $a_n$ is the average remaining uncertainty in the next symbol after its history is known. More history can improve prediction, so this sequence is nonincreasing:
$$
a_{n+1}=H(X_{n+1}\mid X_1,\ldots,X_n)
\leq H(X_{n+1}\mid X_2,\ldots,X_n)=a_n.
$$
The inequality uses <conditioning reduces entropy>; the last equality uses <strict stationarity>. Nonnegative $a_n$ therefore have a <limit>. By the <chain rule for information entropy>,
$$
\frac{H(X_1,\ldots,X_n)}n=\frac1n\sum_{j=1}^n a_j.
$$
The <Cesàro mean> converges to the same <limit>, proving the equality with the <entropy rate>. This is a statement about average <information entropy>. A constant almost-sure limit of the realized <information content> requires the further hypotheses in the <Shannon-McMillan-Breiman theorem>; <stationarity without ergodicity in an information source> explains the distinction.
Back to article page
