= Solution
Let $\xi_0^{n-1}=\bigvee_{j=0}^{n-1}T^{-j}\xi$ denote the <join of measurable partitions> recording the first $n$ observations, and set $H_n=H_\mu(\xi_0^{n-1})$, with $H_0=0$. The <entropy of a countable measurable partition> is $H_\mu(\xi)=-\sum_{A\in\xi}\mu(A)\log\mu(A)$, using natural logarithms and $0\log0=0$. Null atoms can be discarded. Throughout, $\mu$ is a probability measure.
The stronger property needed for <monotonicity of normalized block entropy> is that its increments decrease. Define
$$
c_k=H_\mu\left(\xi\mathrel{\big|}\bigvee_{j=1}^kT^{-j}\xi\right),\qquad c_0=H_\mu(\xi).
$$
Since <conditioning reduces entropy>, $c_{k+1}\leq c_k$. The conditional entropy chain rule and measure preservation give
$$
H_{k+1}
=H_\mu\left(\bigvee_{j=1}^kT^{-j}\xi\right)+c_k
=H_k+c_k,
\qquad H_n=\sum_{k=0}^{n-1}c_k.
$$
It follows that $c_n\leq H_n/n$, and consequently
$$
\boxed{\frac{H_{n+1}}{n+1}=\frac{H_n+c_n}{n+1}\leq\frac{H_n}{n}.}
$$
This uses stationarity as well as the entropy chain rule; subadditivity alone would not establish monotonicity of every successive ratio.
The <entropy rate of a measurable partition> and the <Kolmogorov-Sinai entropy> are, respectively,
$$
\boxed{h_\mu(T,\xi)=\lim_{n\to\infty}\frac1nH_\mu(\xi_0^{n-1})
=\inf_{n\geq1}\frac{H_n}{n},\qquad
h_\mu(T)=\sup_{\eta\text{ finite measurable partition}}h_\mu(T,\eta).}
$$
One may equivalently take the supremum over countable finite-entropy <measurable partitions>.
The <Shannon-McMillan-Breiman theorem> states that for a countable <measurable partition> with $H_\mu(\xi)<\infty$, the normalized information
$$
\frac1nI_\mu(\xi_0^{n-1})(x)
=-\frac1n\log\mu(\xi_0^{n-1}(x))
$$
converges <almost everywhere> and in $L^1$ to a $T$ invariant function $h_\xi(x)$ with integral $h_\mu(T,\xi)$. Here $\xi_0^{n-1}(x)$ is the atom containing $x$. If $T$ is an <ergodic transformation>, then $h_\xi(x)=h_\mu(T,\xi)$ <almost everywhere>. We prove the general form, including an explicit formula for its limit.
Let $\mathcal F_k$ be the <sigma-algebra> generated by $\bigvee_{j=1}^kT^{-j}\xi$, let $\mathcal F_0$ be trivial, and put $\mathcal F_\infty=\bigvee_{j=1}^\infty T^{-j}\sigma(\xi)$. For every atom $A$, the probabilities
$$
p_k^A=\mathbb E_\mu[\mathbf1_A\mid\mathcal F_k]
$$
form a bounded <conditional-expectation martingale>. The <martingale convergence theorem> gives $p_k^A\to p_\infty^A=\mathbb E_\mu[\mathbf1_A\mid\mathcal F_\infty]$ <almost everywhere> and in $L^1$. Each of these probabilities is positive <almost everywhere> on $A$: for example, integrating $\mathbf1_A$ over the $\mathcal F_\infty$ measurable set $\{p_\infty^A=0\}$ gives zero. Countability of $\xi$ lets us choose a common full-measure set for every atom.
The <conditional information functions>
$$
f_k=I_\mu(\xi\mid\mathcal F_k)
=-\sum_{A\in\xi}\mathbf1_A\log p_k^A
$$
therefore converge <almost everywhere> to $f_\infty=I_\mu(\xi\mid\mathcal F_\infty)$. We need an integrable bound on this sequence; convergence of the probabilities alone would not supply one after taking logarithms. The allowed <maximal inequality for conditional information functions> gives, for $F=\sup_{k\geq0}f_k$,
$$
\mu(A\cap\{F>t\})\leq e^{-t}\qquad(A\in\xi,\ t\geq0).
$$
Combining this with the bound by $\mu(A)$ and the <tail integral formula for moments> yields
$$
\int F\,d\mu
\leq\sum_{A\in\xi}\int_0^\infty\min\{\mu(A),e^{-t}\}\,dt
=\sum_{A\in\xi}\mu(A)(1-\log\mu(A))
=1+H_\mu(\xi)<\infty.
$$
Thus $0\leq f_\infty\leq F$, and the <dominated convergence theorem> gives $f_k\to f_\infty$ in $L^1$. In particular,
$$
\int f_\infty\,d\mu=\lim_k c_k=h_\mu(T,\xi),
$$
because $H_n/n$ is the average of the decreasing sequence $(c_k)$.
The information chain rule, applied from the final observation backwards, gives the exact identity
$$
I_\mu(\xi_0^{n-1})(x)
=\sum_{j=0}^{n-1}f_{n-1-j}(T^jx).
$$
Indeed, the symbol at time $j$ is conditioned on times $j+1,\ldots,n-1$; pulling that conditional probability back by $T^j$ gives $f_{n-1-j}\circ T^j$. Measure preservation is sufficient for this pullback identity.
We now prove the required <triangular ergodic averaging lemma> in this instance. Put $g_M=\sup_{k\geq M}|f_k-f_\infty|$. Then $g_M\downarrow0$ <almost everywhere>, $g_M\leq2F$, and $\|g_M\|_1\to0$. Split the difference between the displayed triangular sum and $\sum_{j=0}^{n-1}f_\infty(T^jx)$. Terms whose index is at least $M$ are bounded by $\sum_{j=0}^{n-1}g_M(T^jx)$. The remaining terms are at most
$$
\sum_{k=0}^{M-1}|f_k-f_\infty|(T^{n-1-k}x).
$$
After division by $n$, each of these finitely many end terms tends to zero <almost everywhere> by the <linear growth bound for integrable observables> proved in Question 1. Applying the <Birkhoff ergodic theorem> to $g_M$, for every fixed $M$, therefore gives
$$
\limsup_{n\to\infty}\left|
\frac1n\sum_{j=0}^{n-1}(f_{n-1-j}-f_\infty)(T^jx)
\right|\leq\mathbb E_\mu[g_M\mid\mathcal I](x).
$$
These <conditional expectations> decrease to zero <almost everywhere>: they decrease, and their integrals $\int g_M\,d\mu$ tend to zero. Letting $M\to\infty$, then applying the <Birkhoff ergodic theorem> to $f_\infty$, proves
$$
\boxed{-\frac1n\log\mu(\xi_0^{n-1}(x))
\longrightarrow h_\xi(x)=\mathbb E_\mu[f_\infty\mid\mathcal I](x)
\quad\text{almost everywhere}.}
$$
For $L^1$ convergence, measure preservation gives the direct estimate
$$
\left\|\frac1n\sum_{j=0}^{n-1}(f_{n-1-j}-f_\infty)\circ T^j\right\|_1
\leq\frac1n\sum_{k=0}^{n-1}\|f_k-f_\infty\|_1\longrightarrow0.
$$
Combine this with $L^1$ convergence of the ergodic averages of $f_\infty$. Finally, $\int h_\xi\,d\mu=\int f_\infty\,d\mu=h_\mu(T,\xi)$. In the ergodic case the <invariant sigma-algebra> is trivial, completing the familiar form of the theorem:
$$
\boxed{-\frac1n\log\mu(\xi_0^{n-1}(x))\longrightarrow h_\mu(T,\xi)
\quad\text{almost everywhere and in }L^1.}
$$
Back to article page
