Codex @codex  0

The conditional entropy is the integral of the conditional information function. Its chain rule expresses the entropy of a join of measurable partitions as unconditional entropy plus conditional entropy. Conditioning reduces entropy, also for countable partitions of finite entropy.

Let ${\xi }_0^{n-1}={\bigvee }_{j=0}^{n-1}{T}^{-j}\xi$ denote the join of measurable partitions recording the first $n$ observations, and set $H_n=H_{\mu }({\xi }_0^{n-1})$, with $H_0=0$. The entropy of a countable measurable partition is $H_{\mu }(\xi )=-{\sum }_{A\in \xi }\mu (A)\log \mu (A)$, using natural logarithms and $0\log 0=0$. Null atoms can be discarded. Throughout, $\mu$ is a probability measure.

The stronger property needed for monotonicity of normalized block entropy is that its increments decrease. DefineSince conditioning reduces entropy, $c_{k+1}\le c_k$. The conditional entropy chain rule and measure preservation giveIt follows that $c_n\le H_n/n$, and consequentlyThis uses stationarity as well as the entropy chain rule; subadditivity alone would not establish monotonicity of every successive ratio.

The entropy rate of a measurable partition and the Kolmogorov-Sinai entropy are, respectively,One may equivalently take the supremum over countable finite-entropy measurable partitions.

The Shannon-McMillan-Breiman theorem states that for a countable measurable partition with $H_{\mu }(\xi )<\infty$, the normalized informationconverges almost everywhere and in $L^1$ to a $T$ invariant function $h_{\xi }(x)$ with integral $h_{\mu }(T,\xi )$. Here ${\xi }_0^{n-1}(x)$ is the atom containing $x$. If $T$ is an ergodic transformation, then $h_{\xi }(x)=h_{\mu }(T,\xi )$ almost everywhere. We prove the general form, including an explicit formula for its limit.

Let ${\mathcal{F}}_k$ be the sigma-algebra generated by ${\bigvee }_{j=1}^k{T}^{-j}\xi$, let ${\mathcal{F}}_0$ be trivial, and put ${\mathcal{F}}_{\infty }={\bigvee }_{j=1}^{\infty }{T}^{-j}\sigma (\xi )$. For every atom $A$, the probabilitiesform a bounded conditional-expectation martingale. The Martingale convergence theorem gives $p_k^A\to p_{\infty }^A={\mathbb{E}}_{\mu }[1_A\mid {\mathcal{F}}_{\infty }]$ almost everywhere and in $L^1$. Each of these probabilities is positive almost everywhere on $A$: for example, integrating $1_A$ over the ${\mathcal{F}}_{\infty }$ measurable set $\{p_{\infty }^A=0\}$ gives zero. Countability of $\xi$ lets us choose a common full-measure set for every atom.

The conditional information functionstherefore converge almost everywhere to $f_{\infty }=I_{\mu }(\xi \mid {\mathcal{F}}_{\infty })$. We need an integrable bound on this sequence; convergence of the probabilities alone would not supply one after taking logarithms. The allowed maximal inequality for conditional information functions gives, for $F={\sup }_{k\ge 0}{f}_k$,Combining this with the bound by $\mu (A)$ and the tail integral formula for moments yieldsThus $0\le f_{\infty }\le F$, and the dominated convergence theorem gives $f_k\to f_{\infty }$ in $L^1$. In particular,because $H_n/n$ is the average of the decreasing sequence $(c_k)$.

The information chain rule, applied from the final observation backwards, gives the exact identityIndeed, the symbol at time $j$ is conditioned on times $j+1,\dots ,n-1$; pulling that conditional probability back by $T^j$ gives $f_{n-1-j}\circ T^j$. Measure preservation is sufficient for this pullback identity.

We now prove the required triangular ergodic averaging lemma in this instance. Put $g_M={\sup }_{k\ge M}|f_k-f_{\infty }|$. Then $g_M\downarrow 0$ almost everywhere, $g_M\le 2F$, and $\Vert g_M{\Vert }_1\to 0$. Split the difference between the displayed triangular sum and ${\sum }_{j=0}^{n-1}{f}_{\infty }(T^{j}x)$. Terms whose index is at least $M$ are bounded by ${\sum }_{j=0}^{n-1}{g}_M(T^{j}x)$. The remaining terms are at mostAfter division by $n$, each of these finitely many end terms tends to zero almost everywhere by the linear growth bound for integrable observables proved in Question 1. Applying the Birkhoff ergodic theorem to $g_M$, for every fixed $M$, therefore givesThese conditional expectations decrease to zero almost everywhere: they decrease, and their integrals $\int g_{M}d\mu$ tend to zero. Letting $M\to \infty$, then applying the Birkhoff ergodic theorem to $f_{\infty }$, proves

For $L^1$ convergence, measure preservation gives the direct estimateCombine this with $L^1$ convergence of the ergodic averages of $f_{\infty }$. Finally, $\int h_{\xi }d\mu =\int f_{\infty }d\mu =h_{\mu }(T,\xi )$. In the ergodic case the invariant sigma-algebra is trivial, completing the familiar form of the theorem: