Entropy of a countable measurable partition ($H_{\mu }(\xi )$)

For a countable measurable partition, its entropy is $H_{\mu }(\xi )=-{\sum }_{A\in \xi }\mu (A)\log \mu (A)$, with $0\log 0=0$. This extends entropy of a finite measurable partition and is allowed to be infinite.

For a measurable partition $\xi$ and a sigma-algebra $\mathcal{G}$, the conditional information function isThe conditional expectation in this formula is positive almost everywhere on $A$. Without conditioning, the information is $I_{\mu }(\xi )(x)=-\log \mu (\xi (x))$.

For increasing sigma-algebras ${\mathcal{G}}_n$, put $f_n=I_{\mu }(\xi \mid {\mathcal{G}}_n)$ and $F={\sup }_n{f}_n$. For each atom $A\in \xi$ and $t\ge 0$,Together with the bound by $\mu (A)$ and the tail integral formula for moments, this yields $\int Fd\mu \le H_{\mu }(\xi )+1$. The inequality follows by stopping the conditional-expectation martingale $\mathbb{E}[1_A\mid {\mathcal{G}}_n]$ the first time it falls below $e^{-t}$.

The conditional entropy is the integral of the conditional information function. Its chain rule expresses the entropy of a join of measurable partitions as unconditional entropy plus conditional entropy. Conditioning reduces entropy, also for countable partitions of finite entropy.