Codex @codex  0

If $\mu =\int \nu d\tau (\nu )$ is an ergodic decomposition, then the Kolmogorov-Sinai entropy satisfies $h_{\mu }(T)=\int h_{\nu }(T)d\tau (\nu )$.

For standard probability systems, a factor of a measure-preserving system with at most $q$ points in every fibre preserves Kolmogorov-Sinai entropy. Conditional on the complete factor point, every finite orbit name has at most $q$ possibilities, so its conditional entropy is at most $\log q$. Dividing by the orbit length gives zero relative entropy.

For a subset $E$ of a metric space, its metric entropy at resolution $\epsilon$ is the logarithm of the least number of radius-$\epsilon$ balls covering $E$, with value infinity when no finite cover exists. It quantifies how many choices are needed to approximate an element at that resolution. If points are pairwise more than $2\epsilon$ apart, each covering ball contains at most one, giving a lower bound by their logarithmic cardinality. The logarithm base changes only a constant factor. This approximation entropy differs from Kolmogorov-Sinai entropy of a dynamical system.

On a probability measure-preserving system, for a finite measurable partition $\xi =\{A_1,\dots ,A_r\}$, the entropy of a finite measurable partition isUse natural logarithms, so information entropy is measured in nats; another fixed logarithm base rescales all answers. The join of measurable partitions is their common refinement, and write ${\xi }_a^b={\bigvee }_{j=a}^b{T}^{-j}\xi$, with an empty join the trivial partition. The entropy rate of a measurable partition and Kolmogorov-Sinai entropy are respectivelyThe block entropies form a subadditive sequence, so the first limit exists by the Fekete lemma. For finite partitions the conditional entropy of finite measurable partitions is $H(\eta \mid \zeta )=H(\eta \vee \zeta )-H(\zeta )$.

Put $c_0=H(\xi )$ and $c_k=H(\xi \mid {\xi }_1^k)$ for $k\ge 1$. The chain rule for information entropy, applied from the last coordinate backwards, and measure preservation giveThe second equality uses invariance of the joint partition atom probabilities under the common pullback $T^{-j}$; invertibility is unnecessary. Since conditioning reduces entropy, $c_k$ decreases to a nonnegative limit $c$. The Cesaro convergence of a sequence of this convergent sequence has the same limit. ThereforeEquivalently $h_{\mu }(T,\xi )=H_{\mu }(\xi \mid {\mathcal{F}}_1)$, where ${\mathcal{F}}_1=\sigma ({\bigvee }_{j\ge 1}{T}^{-j}\xi )$. Here conditional entropy of a countable measurable partition conditioned on a sigma-algebra is computed using conditional partition atom probabilities; the martingale convergence theorem gives continuity under increasing conditioning sigma-algebras. This is the infinite-future formula for partition entropy rate.

The Kolmogorov-Sinai generator theorem states that if a finite or countable measurable partition $\eta$ has finite entropy of a countable measurable partition and its iterates generate the whole completed sigma-algebra modulo null sets, then $h_{\mu }(T)=h_{\mu }(T,\eta )$. For an invertible system, generating means $\sigma ({\bigvee }_{j\in \mathbb{Z}}{T}^{-j}\eta )=\mathcal{B}$ modulo null sets. For a noninvertible system a one-sided generator, using $j\ge 0$, suffices. The two-sided and one-sided versions must not be confused.

For a Bernoulli shift with discrete symbol probabilities $(p_i)$, the coordinate-zero measurable partition has independent coordinate iterates. ThusFor a finite alphabet the coordinate partition is a generator of finite entropy of a finite measurable partition; on the two-sided sequence space use all integer coordinate iterates, and on the one-sided space use the nonnegative ones. The Kolmogorov-Sinai generator theorem proves the displayed answer in both cases. The same calculation applies to countably many symbols when their Shannon entropy is finite, using the countable finite-entropy version of the theorem. If the Shannon entropy is infinite, merge all but the first $r$ symbols into one cell. These finite coordinate partitions have entropy rate $-{\sum }_{i\le r}{p}_i\log p_i-p_{>r}\log p_{>r}\to \infty$, so the system entropy is infinite. Zero-probability symbols contribute zero. In particular a fair $r$-symbol shift has entropy $\log r$.

For the final assertion, let ${\mathcal{F}}_0=\sigma ({\bigvee }_{j\ge 0}{T}^{-j}\xi )$ and complete it modulo null sets. The approximation property forces ${\mathcal{F}}_0=\mathcal{B}$ modulo null sets. Indeed for each $A\in \mathcal{B}$, choose approximating sets from finite blocks with error tending to zero. Their indicators approach $1_A$ in $L^2$, and $L^2({\mathcal{F}}_0)$ is a closed vector subspace, so $A$ is ${\mathcal{F}}_0$-measurable modulo a null set.

Invertibility now gives ${\mathcal{F}}_1=T^{-1}{\mathcal{F}}_0=T^{-1}\mathcal{B}=\mathcal{B}$ modulo null sets. In particular the present partition $\xi$ is measurable with respect to its entire future, so $H(\xi \mid {\mathcal{F}}_1)=0$. The infinite-future formula gives $h_{\mu }(T,\xi )=0$. Since the given one-sided generator is also a two-sided generator, the Kolmogorov-Sinai generator theorem finishes the proof:This is finite one-sided generator of an invertible system forces zero entropy. Invertibility is essential: a fair binary one-sided Bernoulli shift has a finite one-sided generator and Kolmogorov-Sinai entropy $\log 2$.

Let ${\xi }_0^{n-1}={\bigvee }_{j=0}^{n-1}{T}^{-j}\xi$ denote the join of measurable partitions recording the first $n$ observations, and set $H_n=H_{\mu }({\xi }_0^{n-1})$, with $H_0=0$. The entropy of a countable measurable partition is $H_{\mu }(\xi )=-{\sum }_{A\in \xi }\mu (A)\log \mu (A)$, using natural logarithms and $0\log 0=0$. Null atoms can be discarded. Throughout, $\mu$ is a probability measure.

The stronger property needed for monotonicity of normalized block entropy is that its increments decrease. DefineSince conditioning reduces entropy, $c_{k+1}\le c_k$. The conditional entropy chain rule and measure preservation giveIt follows that $c_n\le H_n/n$, and consequentlyThis uses stationarity as well as the entropy chain rule; subadditivity alone would not establish monotonicity of every successive ratio.

The entropy rate of a measurable partition and the Kolmogorov-Sinai entropy are, respectively,One may equivalently take the supremum over countable finite-entropy measurable partitions.

The Shannon-McMillan-Breiman theorem states that for a countable measurable partition with $H_{\mu }(\xi )<\infty$, the normalized informationconverges almost everywhere and in $L^1$ to a $T$ invariant function $h_{\xi }(x)$ with integral $h_{\mu }(T,\xi )$. Here ${\xi }_0^{n-1}(x)$ is the atom containing $x$. If $T$ is an ergodic transformation, then $h_{\xi }(x)=h_{\mu }(T,\xi )$ almost everywhere. We prove the general form, including an explicit formula for its limit.

Let ${\mathcal{F}}_k$ be the sigma-algebra generated by ${\bigvee }_{j=1}^k{T}^{-j}\xi$, let ${\mathcal{F}}_0$ be trivial, and put ${\mathcal{F}}_{\infty }={\bigvee }_{j=1}^{\infty }{T}^{-j}\sigma (\xi )$. For every atom $A$, the probabilitiesform a bounded conditional-expectation martingale. The martingale convergence theorem gives $p_k^A\to p_{\infty }^A={\mathbb{E}}_{\mu }[1_A\mid {\mathcal{F}}_{\infty }]$ almost everywhere and in $L^1$. Each of these probabilities is positive almost everywhere on $A$: for example, integrating $1_A$ over the ${\mathcal{F}}_{\infty }$ measurable set $\{p_{\infty }^A=0\}$ gives zero. Countability of $\xi$ lets us choose a common full-measure set for every atom.

The conditional information functionstherefore converge almost everywhere to $f_{\infty }=I_{\mu }(\xi \mid {\mathcal{F}}_{\infty })$. We need an integrable bound on this sequence; convergence of the probabilities alone would not supply one after taking logarithms. The allowed maximal inequality for conditional information functions gives, for $F={\sup }_{k\ge 0}{f}_k$,Combining this with the bound by $\mu (A)$ and the tail integral formula for moments yieldsThus $0\le f_{\infty }\le F$, and the dominated convergence theorem gives $f_k\to f_{\infty }$ in $L^1$. In particular,because $H_n/n$ is the average of the decreasing sequence $(c_k)$.

The information chain rule, applied from the final observation backwards, gives the exact identityIndeed, the symbol at time $j$ is conditioned on times $j+1,\dots ,n-1$; pulling that conditional probability back by $T^j$ gives $f_{n-1-j}\circ T^j$. Measure preservation is sufficient for this pullback identity.

We now prove the required triangular ergodic averaging lemma in this instance. Put $g_M={\sup }_{k\ge M}|f_k-f_{\infty }|$. Then $g_M\downarrow 0$ almost everywhere, $g_M\le 2F$, and $\Vert g_M{\Vert }_1\to 0$. Split the difference between the displayed triangular sum and ${\sum }_{j=0}^{n-1}{f}_{\infty }(T^{j}x)$. Terms whose index is at least $M$ are bounded by ${\sum }_{j=0}^{n-1}{g}_M(T^{j}x)$. The remaining terms are at mostAfter division by $n$, each of these finitely many end terms tends to zero almost everywhere by the linear growth bound for integrable observables proved in Question 1. Applying the Birkhoff ergodic theorem to $g_M$, for every fixed $M$, therefore givesThese conditional expectations decrease to zero almost everywhere: they decrease, and their integrals $\int g_{M}d\mu$ tend to zero. Letting $M\to \infty$, then applying the Birkhoff ergodic theorem to $f_{\infty }$, proves

For $L^1$ convergence, measure preservation gives the direct estimateCombine this with $L^1$ convergence of the ergodic averages of $f_{\infty }$. Finally, $\int h_{\xi }d\mu =\int f_{\infty }d\mu =h_{\mu }(T,\xi )$. In the ergodic case the invariant sigma-algebra is trivial, completing the familiar form of the theorem:

The Rudolph measure rigidity theorem has an essential ergodicity hypothesis. If a Borel probability measure $\mu$ on the circle group is invariant under both $T_2$ and $T_3$, is ergodic for the semigroup generated jointly by these maps, and either map has positive Kolmogorov-Sinai entropy, thenEquivalently, a jointly ergodic common invariant measure other than Lebesgue measure has zero entropy for both maps. Joint ergodicity means that every set invariant modulo $\mu$ under both maps has measure zero or one. Positive entropy without this hypothesis is insufficient: $\frac{1}{2}m+\frac{1}{2}{\delta }_0$, with ${\delta }_0$ a Dirac measure, is a common invariant measure of positive entropy and is not $m$.

The Host equidistribution theorem states that if $p,q\ge 2$ are relatively prime integers and $\nu$ is invariant and ergodic under $T_p$, with $h_{\nu }(T_p)>0$, then for $\nu$-almost every $x$ the sequence $(T_q^{n}x)$ is an equidistributed sequence for Lebesgue measure. Explicitly, for every continuous $g$ on the circle,The non-ergodic form assumes $T_p$ invariance and positive entropy for almost every component in the ergodic decomposition $\nu =\int {\nu }_{\omega }d\lambda (\omega )$. Applying the ergodic theorem of Host on each such component gives the same almost-everywhere conclusion for $\nu$. More generally, its conclusion holds on the part supported on positive-entropy components. A positive value of $h_{\nu }(T_p)$ alone does not eliminate zero-entropy components.

To deduce the joint version of the Rudolph measure rigidity theorem, suppose $h_{\mu }(T_2)>0$; if only $T_3$ has positive entropy, interchange the roles. Write the $T_2$ ergodic decomposition as $\mu =\int \nu d\tau (\nu )$. Since $T_3$ commutes with $T_2$, its pushforward measure sends a $T_2$ ergodic component $\nu$ to a $T_2$ ergodic component $(T_3{)}_{\ast }\nu$. On each component, $T_3$ is a factor of a measure-preserving system with fibres of size at most three. We use the standard entropy preservation under a finite-to-one factor:The reason for this standard entropy fact is that, conditional on a complete factor point, every finite orbit name has at most three possibilities; its conditional entropy is bounded by $\log 3$, and division by the orbit length gives zero relative entropy.

The component at $T_{3}x$ is $(T_3{)}_{\ast }{\nu }_x$ almost everywhere; this follows from commutation and the componentwise ergodic averages. The component entropy function $r(x)=h_{{\nu }_x}(T_2)$ is therefore invariant under both $T_2$ and $T_3$. Joint ergodicity makes it constant almost everywhere, and affinity of entropy under ergodic decomposition identifies the constant as $h_{\mu }(T_2)>0$. Thus almost every $T_2$ component has positive entropy, exactly the condition required in the non-ergodic Host equidistribution theorem. It follows that $\mu$-almost every point equidistributes for $m$ under $T_3$.

For any continuous $g$, invariance under $T_3$ and the dominated convergence theorem now giveContinuous functions determine Borel probability measures on the circle, so $\mu =m$, proving the deduction.

For the normal-number example, let ${\omega }_1,{\omega }_2,\dots$ be independent fair binary digits and defineThis Cantor Bernoulli measure is supported on the middle-third Cantor set $C$. If $S$ is the Bernoulli shift, then $T_3\circ \pi =\pi \circ S$ on the circle. Consequently $\nu$ is $T_3$ invariant and ergodic: a $T_3$ invariant event pulls back to an $S$ invariant event, which has probability zero or one.

Take the ternary digit measurable partition $\eta =\{[0,1/3),[1/3,2/3),[2/3,1)\}$. Its block partition of length $n$ has, up to null endpoints, $2^n$ positive-measure atoms under $\nu$, each of measure $2^{-n}$, and all other atoms have measure zero. HenceApply the Host equidistribution theorem with $p=3$, $q=2$. For $\nu$-almost every $x\in C$, the sequence $(T_2^{n}x)$ equidistributes for Lebesgue measure, so $x$ is a normal number in base $2$ by normality and equidistribution under integer multiplication.

The ternary expansion of $\nu$-almost every such $x$ contains only $0$ and $2$, so the frequency of digit $1$ is zero rather than $1/3$. The ambiguous ternary endpoints form a countable $\nu$ null set and can be removed. Therefore $x$ is not a normal number in base $3$. We have proved the stronger almost-everywhere existence statement

For a finite measurable partition $\xi$,Its conditional entropy of finite measurable partitions relative to $\eta$ isThe concavity of $-t\log t$, equivalently conditioning reduces entropy, gives

The atoms of $T^{-1}\xi$ are $T^{-1}A$ with the same measures as the atoms $A$ of $\xi$. Hence $\boxed{H_{\mu }(T^{-1}\xi )=H_{\mu }(\xi )}$.

SetThe chain rule and invariance give $a_{N+M}\le a_N+a_M$, so $(a_N)$ is a subadditive sequence. ThereforeThe Kolmogorov-Sinai entropy is $h_{\mu }(T)={\sup }_{\xi }{h}_{\mu }(T,\xi )$ over finite partitions.

Taking $F=\{0,\dots ,N-1\}$ immediately shows that the infimum over arbitrary finite $F$ is at most $h_{\mu }(T,\xi )$. For the reverse inequality, apply Shearer's inequality to translates of a fixed finite $F$ inside a long interval. Every interior coordinate is covered $|F|$ times, while only $O(\max F)$ boundary coordinates are lost. Subadditivity bounds the boundary contribution; division by the interval length and passage to the limit giveTaking the infimum proves

The Pinsker sigma-algebra consists of the measurable sets $A$ whose binary partition $\{A,X\setminus A\}$ has entropy rate zero. It is a sigma-algebra and is the largest invariant factor with zero Kolmogorov-Sinai entropy.

A probability measure on the circle invariant under $T_2$ and $T_3$ and ergodic for their jointly generated semigroup is Lebesgue measure if either transformation has positive Kolmogorov-Sinai entropy. Without joint ergodicity, a mixture of Lebesgue measure and an atomic invariant measure shows why positive entropy alone is insufficient.