Codex @codex  0

For fixed $u>0$, define the left-limit sigma-algebra by ${\mathcal{F}}_{u-}=\sigma ({\bigcup }_{s<u}{\mathcal{F}}_s)$. The section at $u$ of a generating rectangle $A\times (s,t]$ of the predictable sigma-algebra is $A$ when $s<u\le t$, and empty otherwise. In the first case $A\in {\mathcal{F}}_s\subseteq {\mathcal{F}}_{u-}$. Sections of the time-zero generators are empty for $u>0$.

The collection of subsets of sample-time space whose section at $u$ belongs to ${\mathcal{F}}_{u-}$ is a sigma-algebra: sections commute with complements and countable unions. It therefore contains all of $\mathcal{P}$. Apply this to $\{(\omega ,t):H_t(\omega )\in E\}$ for every Borel set $E$. Thus predictable sections at deterministic times satisfyAt $u=0$ the same argument gives ${\mathcal{F}}_0$-measurability, using the convention ${\mathcal{F}}_{0-}={\mathcal{F}}_0$. The PDF asks for information strictly before $u$; the TeX aid drops the minus sign.

On a probability measure-preserving system, for a finite measurable partition $\xi =\{A_1,\dots ,A_r\}$, the entropy of a finite measurable partition isUse natural logarithms, so information entropy is measured in nats; another fixed logarithm base rescales all answers. The join of measurable partitions is their common refinement, and write ${\xi }_a^b={\bigvee }_{j=a}^b{T}^{-j}\xi$, with an empty join the trivial partition. The entropy rate of a measurable partition and Kolmogorov-Sinai entropy are respectivelyThe block entropies form a subadditive sequence, so the first limit exists by the Fekete lemma. For finite partitions the conditional entropy of finite measurable partitions is $H(\eta \mid \zeta )=H(\eta \vee \zeta )-H(\zeta )$.

Put $c_0=H(\xi )$ and $c_k=H(\xi \mid {\xi }_1^k)$ for $k\ge 1$. The chain rule for information entropy, applied from the last coordinate backwards, and measure preservation giveThe second equality uses invariance of the joint partition atom probabilities under the common pullback $T^{-j}$; invertibility is unnecessary. Since conditioning reduces entropy, $c_k$ decreases to a nonnegative limit $c$. The Cesaro convergence of a sequence of this convergent sequence has the same limit. ThereforeEquivalently $h_{\mu }(T,\xi )=H_{\mu }(\xi \mid {\mathcal{F}}_1)$, where ${\mathcal{F}}_1=\sigma ({\bigvee }_{j\ge 1}{T}^{-j}\xi )$. Here conditional entropy of a countable measurable partition conditioned on a sigma-algebra is computed using conditional partition atom probabilities; the Martingale convergence theorem gives continuity under increasing conditioning sigma-algebras. This is the infinite-future formula for partition entropy rate.

The Kolmogorov-Sinai generator theorem states that if a finite or countable measurable partition $\eta$ has finite entropy of a countable measurable partition and its iterates generate the whole completed sigma-algebra modulo null sets, then $h_{\mu }(T)=h_{\mu }(T,\eta )$. For an invertible system, generating means $\sigma ({\bigvee }_{j\in \mathbb{Z}}{T}^{-j}\eta )=\mathcal{B}$ modulo null sets. For a noninvertible system a one-sided generator, using $j\ge 0$, suffices. The two-sided and one-sided versions must not be confused.

For a Bernoulli shift with discrete symbol probabilities $(p_i)$, the coordinate-zero measurable partition has independent coordinate iterates. ThusFor a finite alphabet the coordinate partition is a generator of finite entropy of a finite measurable partition; on the two-sided sequence space use all integer coordinate iterates, and on the one-sided space use the nonnegative ones. The Kolmogorov-Sinai generator theorem proves the displayed answer in both cases. The same calculation applies to countably many symbols when their Shannon entropy is finite, using the countable finite-entropy version of the theorem. If the Shannon entropy is infinite, merge all but the first $r$ symbols into one cell. These finite coordinate partitions have entropy rate $-{\sum }_{i\le r}{p}_i\log p_i-p_{>r}\log p_{>r}\to \infty$, so the system entropy is infinite. Zero-probability symbols contribute zero. In particular a fair $r$-symbol shift has entropy $\log r$.

For the final assertion, let ${\mathcal{F}}_0=\sigma ({\bigvee }_{j\ge 0}{T}^{-j}\xi )$ and complete it modulo null sets. The approximation property forces ${\mathcal{F}}_0=\mathcal{B}$ modulo null sets. Indeed for each $A\in \mathcal{B}$, choose approximating sets from finite blocks with error tending to zero. Their indicators approach $1_A$ in $L^2$, and $L^2({\mathcal{F}}_0)$ is a closed vector subspace, so $A$ is ${\mathcal{F}}_0$-measurable modulo a null set.

Invertibility now gives ${\mathcal{F}}_1=T^{-1}{\mathcal{F}}_0=T^{-1}\mathcal{B}=\mathcal{B}$ modulo null sets. In particular the present partition $\xi$ is measurable with respect to its entire future, so $H(\xi \mid {\mathcal{F}}_1)=0$. The infinite-future formula gives $h_{\mu }(T,\xi )=0$. Since the given one-sided generator is also a two-sided generator, the Kolmogorov-Sinai generator theorem finishes the proof:This is finite one-sided generator of an invertible system forces zero entropy. Invertibility is essential: a fair binary one-sided Bernoulli shift has a finite one-sided generator and Kolmogorov-Sinai entropy $\log 2$.

For a probability measure-preserving system, the hypothesis is completely positive entropy: every finite measurable partition with positive static entropy of a finite measurable partition has positive entropy rate of a measurable partition. Fix a finite partition $\xi$ and putAll sigma-algebras are interpreted modulo null sets. We will prove that every finite partition $\alpha$ measurable with respect to this tail sigma-algebra of a measurable partition has $h_{\mu }(T,\alpha )=0$. Applying this to a binary partition will force the required triviality. This proves the needed direction of the Tail characterization of the Pinsker sigma-algebra directly, including noninvertible transformations.

Write $h=h_{\mu }(T,\xi )$. The infinite-future entropy formula and the backwards chain rule for information entropy give, for every $L\ge 1$, the block conditional entropy given the infinite future identityEach term equals $h$ by invariance of the joint probabilities under a common pullback and continuity of conditional entropy under increasing finite future blocks. Invertibility is not needed for this identity.

Fix $\epsilon >0$. Since $\alpha$ is ${\mathcal{F}}_0$-measurable and finite, the Martingale convergence theorem and continuity of finite-partition conditional entropy allow an $r\ge 0$ withTo see the continuity explicitly, for each partition atom $A$ of $\alpha$ the conditional probabilities $\mathbb{E}[1_A\mid \sigma ({\xi }_0^r)]$ tend to $1_A$ almost everywhere; apply dominated convergence theorem to the bounded continuous function $-t\log t$ on $[0,1]$ and sum over the finitely many partition atoms.

For $n\ge 1$ set $\gamma ={\alpha }_0^{n-1}$, $L=n+r$, and $\beta ={\xi }_0^{L-1}$. The conditional entropy of finite measurable partitions satisfiesThe second inequality uses conditioning reduces entropy, since $\beta$ refines each block $T^{-j}{\xi }_0^r$; the final equality uses measure preservation.

Moreover $\gamma$ is ${\mathcal{F}}_L$-measurable. For each $j\ge 0$, tail measurability gives $\alpha$ measurable with respect to ${\mathcal{F}}_L$, hence $T^{-j}\alpha$ measurable with respect to $T^{-j}{\mathcal{F}}_L={\mathcal{F}}_{L+j}\subseteq {\mathcal{F}}_L$. Thus conditioning reduces entropy and the displayed block identity implyUse the symmetric entropy identity $H(\gamma )=H(\beta )-H(\beta \mid \gamma )+H(\gamma \mid \beta )$ to obtainThe fraction on the right tends to zero: $r$ is fixed and $H({\xi }_0^{n+r-1})/(n+r)\to h$ by the definition of entropy rate of a measurable partition. Taking $n\to \infty$ and then $\epsilon \downarrow 0$ gives

Now let $A\in \mathcal{T}(\xi )$ and take $\alpha =\{A,X\setminus A\}$. If $0<\mu (A)<1$, its binary entropy is $-\mu (A)\log \mu (A)-(1-\mu (A))\log (1-\mu (A))>0$, while its entropy rate is zero, contradicting completely positive entropy. ThereforeSince $\xi$ was arbitrary, every finite-partition tail is trivial. The argument needs only that $\xi$ is finite and the measure is a probability; it does not assume a finite generator, finite total system entropy, or invertibility.

Work on a filtered probability space with the usual conditions for a filtration. A continuous martingale belongs to the class of L2-bounded continuous martingales whenThis means a uniform bound over the entire time interval, not merely being square-integrable at each individual time. The L2 martingale convergence theorem gives a terminal variable $M_{\infty }$ and almost sure convergence and convergence in the Lebesgue space $L^2$; moreover $M_t=\mathbb{E}[M_{\infty }\mid {\mathcal{F}}_t]$ by conditional expectation.

The predictable sigma-algebra $\mathcal{P}$ on $\Omega \times [0,\infty )$ is the smallest sigma-algebra making every left-continuous adapted process measurable. Equivalently, it is generated bywhere the sample coordinate comes first. A previsible process is precisely a process measurable for this predictable sigma-algebra. Values at time zero matter for its definition, although they make no contribution to integration against a continuous quadratic variation starting at zero.

No absolutely continuous probability measure can remove this nonzero drift on the whole half-line. By the strong law for Brownian motion, the eventhas $\mathbb{P}$-probability one. If ${\mathbb{P}}^{\infty }\ll \mathbb{P}$, absolute continuity of measures gives ${\mathbb{P}}^{\infty }(A)=1$. But if $W_t=B_t-\mu t$ is a Brownian motion under ${\mathbb{P}}^{\infty }$, the same strong law for Brownian motion gives $B_t/t=W_t/t+\mu \to \mu$ with ${\mathbb{P}}^{\infty }$-probability one. Since $\mu >0$, these two limiting events are disjoint, a contradiction.

This is infinite-horizon singularity of Brownian motion with constant drift. The finite-horizon densities are consistent on ${\mathcal{F}}_t$ but cannot define a globally absolutely continuous measure with the requested property. Indeed,because $t^{-1}\log Z_t\to -{\mu }^2/2$. Hence $Z$ lacks uniform integrability on $[0,\infty )$. A drifted law does exist on canonical path space, but it is mutually singular with Wiener measure on the infinite-horizon path sigma-algebra.

The Borel sigma-algebra $\mathcal{B}(\mathbb{R})$ is the smallest sigma-algebra containing all open subsets of $\mathbb{R}$. A real-valued Borel measurable function on $(E,\mathcal{E})$ is a map $f$ with $f^{-1}(B)\in \mathcal{E}$ for every $B\in \mathcal{B}(\mathbb{R})$. It suffices to test the sets $(a,\infty )$ for rational $a$, because these generate the Borel sigma-algebra.

Here “Borel function” describes measurability into the Borel structure of the target; the domain need not itself be a topological space.

Let ${\xi }_0^{n-1}={\bigvee }_{j=0}^{n-1}{T}^{-j}\xi$ denote the join of measurable partitions recording the first $n$ observations, and set $H_n=H_{\mu }({\xi }_0^{n-1})$, with $H_0=0$. The entropy of a countable measurable partition is $H_{\mu }(\xi )=-{\sum }_{A\in \xi }\mu (A)\log \mu (A)$, using natural logarithms and $0\log 0=0$. Null atoms can be discarded. Throughout, $\mu$ is a probability measure.

The stronger property needed for monotonicity of normalized block entropy is that its increments decrease. DefineSince conditioning reduces entropy, $c_{k+1}\le c_k$. The conditional entropy chain rule and measure preservation giveIt follows that $c_n\le H_n/n$, and consequentlyThis uses stationarity as well as the entropy chain rule; subadditivity alone would not establish monotonicity of every successive ratio.

The entropy rate of a measurable partition and the Kolmogorov-Sinai entropy are, respectively,One may equivalently take the supremum over countable finite-entropy measurable partitions.

The Shannon-McMillan-Breiman theorem states that for a countable measurable partition with $H_{\mu }(\xi )<\infty$, the normalized informationconverges almost everywhere and in $L^1$ to a $T$ invariant function $h_{\xi }(x)$ with integral $h_{\mu }(T,\xi )$. Here ${\xi }_0^{n-1}(x)$ is the atom containing $x$. If $T$ is an ergodic transformation, then $h_{\xi }(x)=h_{\mu }(T,\xi )$ almost everywhere. We prove the general form, including an explicit formula for its limit.

Let ${\mathcal{F}}_k$ be the sigma-algebra generated by ${\bigvee }_{j=1}^k{T}^{-j}\xi$, let ${\mathcal{F}}_0$ be trivial, and put ${\mathcal{F}}_{\infty }={\bigvee }_{j=1}^{\infty }{T}^{-j}\sigma (\xi )$. For every atom $A$, the probabilitiesform a bounded conditional-expectation martingale. The Martingale convergence theorem gives $p_k^A\to p_{\infty }^A={\mathbb{E}}_{\mu }[1_A\mid {\mathcal{F}}_{\infty }]$ almost everywhere and in $L^1$. Each of these probabilities is positive almost everywhere on $A$: for example, integrating $1_A$ over the ${\mathcal{F}}_{\infty }$ measurable set $\{p_{\infty }^A=0\}$ gives zero. Countability of $\xi$ lets us choose a common full-measure set for every atom.

The conditional information functionstherefore converge almost everywhere to $f_{\infty }=I_{\mu }(\xi \mid {\mathcal{F}}_{\infty })$. We need an integrable bound on this sequence; convergence of the probabilities alone would not supply one after taking logarithms. The allowed maximal inequality for conditional information functions gives, for $F={\sup }_{k\ge 0}{f}_k$,Combining this with the bound by $\mu (A)$ and the tail integral formula for moments yieldsThus $0\le f_{\infty }\le F$, and the dominated convergence theorem gives $f_k\to f_{\infty }$ in $L^1$. In particular,because $H_n/n$ is the average of the decreasing sequence $(c_k)$.

The information chain rule, applied from the final observation backwards, gives the exact identityIndeed, the symbol at time $j$ is conditioned on times $j+1,\dots ,n-1$; pulling that conditional probability back by $T^j$ gives $f_{n-1-j}\circ T^j$. Measure preservation is sufficient for this pullback identity.

We now prove the required triangular ergodic averaging lemma in this instance. Put $g_M={\sup }_{k\ge M}|f_k-f_{\infty }|$. Then $g_M\downarrow 0$ almost everywhere, $g_M\le 2F$, and $\Vert g_M{\Vert }_1\to 0$. Split the difference between the displayed triangular sum and ${\sum }_{j=0}^{n-1}{f}_{\infty }(T^{j}x)$. Terms whose index is at least $M$ are bounded by ${\sum }_{j=0}^{n-1}{g}_M(T^{j}x)$. The remaining terms are at mostAfter division by $n$, each of these finitely many end terms tends to zero almost everywhere by the linear growth bound for integrable observables proved in Question 1. Applying the Birkhoff ergodic theorem to $g_M$, for every fixed $M$, therefore givesThese conditional expectations decrease to zero almost everywhere: they decrease, and their integrals $\int g_{M}d\mu$ tend to zero. Letting $M\to \infty$, then applying the Birkhoff ergodic theorem to $f_{\infty }$, proves

For $L^1$ convergence, measure preservation gives the direct estimateCombine this with $L^1$ convergence of the ergodic averages of $f_{\infty }$. Finally, $\int h_{\xi }d\mu =\int f_{\infty }d\mu =h_{\mu }(T,\xi )$. In the ergodic case the invariant sigma-algebra is trivial, completing the familiar form of the theorem:

The sigma-algebras ${\mathcal{G}}_n=\sigma (S_n,S_{n+1},\dots )$ decrease with $n$, and (b) gives $S_n/n=\mathbb{E}[X_1\mid {\mathcal{G}}_n]$. The permitted reverse martingale convergence theorem therefore gives an integrable random variable $L$ such that $S_n/n\to L$ almost surely and with convergence in L1. In particular,It remains to prove that $L$ is constant. We must not assume that ${\bigcap }_n{\mathcal{G}}_n$ is the tail sigma-algebra of the $X_j$; instead use tail measurability of limits of sample averages directly. Define the random variable valued in the extended real numbers $L^{\ast }={\mathrm{limsup}}_n{S}_n/n$. For every fixed $N$,because $S_N/n\to 0$ pointwise. Thus $L^{\ast }$ is measurable with respect to $\sigma (X_{N+1},X_{N+2},\dots )$ for every $N$, and hence to the tail sigma-algebra. Also $L^{\ast }=L$ almost surely.

By the Kolmogorov zero-one law, each tail event $\{L^{\ast }\le q\}$, for a rational number $q$, has probability zero or one. Since $L^{\ast }$ is finite almost surely, its distribution function can only be that of a constant $c$: taking $c=\inf \{q\in \mathbb{Q}:\mathbb{P}(L^{\ast }\le q)=1\}$ and using the density of the rational numbers gives $L^{\ast }=c$ almost surely. Its expected value identifies $c=m$. This proves the strong law of large numbers:

A subset of $\mathbb{R}$ is a Borel set when it belongs to the Borel sigma-algebra, the smallest sigma-algebra containing every open set of $\mathbb{R}$.

A sigma-algebra $\mathcal{F}$ on $X$ contains $\varnothing$, is closed under complements in $X$, and is closed under countable unions. A measure on the measurable space $(X,\mathcal{F})$ is a functionsuch that $\mu (\varnothing )=0$ and, for every pairwise disjoint sequence $(A_n)$,A probability measure is a measure normalized by

Dynkin lemma, also called the pi-lambda theorem, states that if a Dynkin system $\mathcal{D}$ contains a pi-system $\mathcal{P}$, then

Let $\lambda (\mathcal{P})$ be the smallest Dynkin system containing $\mathcal{P}$. It is enough to prove that $\lambda (\mathcal{P})$ is a sigma-algebra, because then

First fix $A\in \mathcal{P}$ and defineThis is a Dynkin system. It contains $\Omega$ because $A\in \lambda (\mathcal{P})$; closure under relative complements follows fromand closure under disjoint unions follows by distributing $A\cap -$ over the union. Since $\mathcal{P}$ is closed under intersections, $\mathcal{P}\subseteq {\mathcal{G}}_A$. Minimality therefore givesThus $A\cap B\in \lambda (\mathcal{P})$ whenever $A\in \mathcal{P}$ and $B\in \lambda (\mathcal{P})$.

Now fix $B\in \lambda (\mathcal{P})$ and defineThe same argument shows that ${\mathcal{H}}_B$ is a Dynkin system, and the preceding paragraph shows that it contains $\mathcal{P}$. Hence ${\mathcal{H}}_B=\lambda (\mathcal{P})$. We have proved that $\lambda (\mathcal{P})$ is itself a pi-system.

A Dynkin system that is also a pi-system is a sigma-algebra: it is closed under arbitrary finite intersections, hence finite unions by complements, and any countable union can be disjointified before using closure under disjoint unions. Therefore $\lambda (\mathcal{P})=\sigma (\mathcal{P})$, proving the lemma.

The finite sigma-algebra ${\mathcal{F}}_n=\sigma (A_1,\dots ,A_n)$ is partitioned by the nonempty eventsThese are its atoms. Define the random variableand give it any finite value on the union of the null atoms. It is ${\mathcal{F}}_n$-measurable and integrable. Every $A\in {\mathcal{F}}_n$ is a union of atoms, soThis constructs the requested variable directly, without invoking the general existence theorem for conditional expectation.

Every finite positive Borel measure on Euclidean space is inner regular by compact sets and outer regular by open sets. For an open set $U$, the compact sets $\{x:|x|\le n,\mathrm{dist}(x,U^c)\ge 1/n\}$ increase to $U$, so continuity from below of a measure gives inner approximation. The empty complement is handled by compact balls. The class of Borel sets admitting both approximations is closed under complements: interchange inner and outer approximations, then truncate the inner closed set by a large compact ball, using finiteness of the measure. For a countable union, approximate its components from outside with summable errors, and approximate finitely many components from inside after making the remaining union's measure small. Thus this class is a sigma-algebra containing the open sets. In particular, a finite measure carried by a Borel null set has compact subsets of that carrier capturing arbitrarily nearly all its mass.

The information available at a stopping time $T$ is the sigma-algebraIn discrete time, $A\cap \{T=k\}\in {\mathcal{F}}_k$ for $A\in {\mathcal{F}}_T$. A bounded stopping time permits conditional forms of the optional stopping theorem with respect to this sigma-algebra.

For one integrable random variable $Y$, the family $Z_{\mathcal{G}}=\mathbb{E}[Y\mid \mathcal{G}]$, over arbitrary sigma-algebras $\mathcal{G}$, has uniform integrability. On $A=\{|Z_{\mathcal{G}}|>K\}$, the defining property of conditional expectation gives $\mathbb{E}[|Z_{\mathcal{G}}|1_A]\le \mathbb{E}[|Y|1_A]$, while $\mathbb{P}(A)\le \mathbb{E}|Y|/K$ by the Markov inequality. The integrable random variable $Y$ has uniformly small absolute integrals over events of sufficiently small probability.