Codex @codex  0

For discrete random variables, write $p(u,v)$ for the joint probability mass function and $p_U(u)$ for its marginal. The conditional entropy is the average of the entropies of the conditional distributions:Conditional distributions at zero-probability values of $u$ can be chosen arbitrarily; they contribute nothing. Here $h$ denotes Shannon entropy, with one common logarithm base and $0\log 0=0$.

For $p(u,v)>0$, use $\log p(u,v)=\log p_U(u)+\log p(v\mid u)$ and sum with weight $-p(u,v)$. Summing out $v$ in the first term givesFor countable alphabets, the entropy contributions are nonnegative, so the equality remains valid with extended values by summing nonnegative terms. Applying this identity successively to the prefix $(X_1,\dots ,X_{i-1})$ and $X_i$ proves the chain rule for information entropy:For $i=1$, the conditioning tuple is empty and the term is $h(X_1)$.

For the subset inequalities, first suppose the marginal entropies are finite, so all joint entropies below are finite by the chain rule and the assumed conditioning reduces entropy property. Put $H=h(X_1,\dots ,X_n)$, and let $H_{-i}$ be the joint entropy with coordinate $i$ omitted. Apply the two-variable chain rule in the order $(X_{[n]\setminus \{i\}},X_i)$ to getThe inequality holds because the prefix conditioning set is contained in the set of all other coordinates. Summing over $i$ and using the chain rule givesThis proves Han's entropy inequality. In terms of the normalized subset entropy, $h_n^{(n)}=H/n$ and $h_{n-1}^{(n)}={\sum }_i{H}_{-i}/[n(n-1)]$, hence $\boxed{h_n^{(n)}\le h_{n-1}^{(n)}}$ for $n\ge 2$.

Apply the same result to each $k$-element coordinate set $S$. It givesWhen this is summed over all $k$-subsets, each fixed $(k-1)$-subset $T$ occurs once for each possible added coordinate, namely $n-k+1$ times. ThereforeSubstitute ${\sum }_{|S|=k}h(X_S)=k\left(\genfrac{}{}{0ex}{}{n}{k}\right)h_k^{(n)}$ and the analogous identity at $k-1$. The binomial coefficient identity $k\left(\genfrac{}{}{0ex}{}{n}{k}\right)=(n-k+1)\left(\genfrac{}{}{0ex}{}{n}{k-1}\right)$ cancels the common positive factor and givesThis is the monotonicity of normalized subset entropies, obtained from the local $k$-coordinate inequality by explicit counting.

If some marginal entropy is infinite, each subset average contains at least one subset including that coordinate. The entropy of such a subset is at least that marginal entropy, by the nonnegative conditional-entropy chain rule, so every $h_k^{(n)}$ is infinite. The inequalities then hold in the extended sense. Otherwise the finite-entropy proof applies. For independent random variables the averages are all $n^{-1}{\sum }_{i}h(X_i)$; repeated copies of one variable instead give $h_k^{(n)}=h(X_1)/k$, illustrating the entropy reduction caused by redundancy.

The Balister-Bollobas inequality says that if $r_i$ is the coordinate multiplicity in a multiset $\mathcal{A}$, and $L_j=\{i:r_i\ge j\}$, then ${\sum }_{j}H(X_{L_j})\le {\sum }_{A\in \mathcal{A}}H(X_A)$. More generally, each union-intersection compression decreases the entropy sum, by entropy submodularity.

For an exact $k$-cover define $A^{-}=[\min A-1]$ and $A^{+}=[\max A-1]\setminus A$. Compressing the family of $A\cup A^{-}$ gives the nested family of all $A^{-}$ together with $k$ full sets. Rearranging the resulting bound gives the upper Madiman-Tetali inequality. Conversely, compressing the family of all $A^{+}$ together with $k$ full sets gives the nested family of $A\cup A^{+}$, yielding the lower bound. In conditional-entropy notation these areThe coordinate multiplicities agree in each compression, because $\mathcal{A}$ is an exact $k$-cover. An equivalent proof expands the chain rule for information entropy contributions $h_i=H(X_i\mid X_{[i-1]})$ and uses conditioning reduces entropy. It also gives the upper inequality for arbitrary fractional covers and the lower inequality for fractional packings.

Let $i(G)$ be the number of independent sets of the biregular graph $G$. Choose one such set $I$ uniformly and write $X_v=1_{v\in I}$. All information entropies below use logarithms to base two, soWe will prove $\boxed{i(G)\le (2^r+2^s-1{)}^{m/s}}$.

We state and justify the entropy facts used. For a finite-valued random variable $Y$, $H(Y)=-{\sum }_y{p}_y{\log }_2{p}_y$ with $0{\log }_{2}0=0$. Its conditional entropy is the mean of this quantity over the conditioning variable. Factoring the joint probabilities proves the chain rule for information entropy, $H(Y,Z)=H(Y)+H(Z\mid Y)$. Conditioning reduces entropy: $H(Y\mid Z,T)\le H(Y\mid Z)$. To see this, condition on $Z$ and use concavity of $-x{\log }_{2}x$ to compare the mixture of the conditional distributions given $T$ with their mean entropy.

The precise Shearer inequality needed here is: if subsets $S_j$ of the indices of a finite random vector $Y$ cover every index at least $d$ times, then $dH(Y)\le {\sum }_{j}H(Y_{S_j})$. To prove it, order the indices. The chain rule for information entropy expandsThe inequality is conditioning reduces entropy. Summing over $j$ counts each nonnegative conditional term at least $d$ times and proves the assertion.

For every $w\in W$, its vertex neighbourhood $N(w)\subseteq U$ has $s$ vertices. Every $u\in U$ lies in exactly $r$ of these neighbourhoods. Hence Shearer inequality givesLet $q_w=\mathbb{P}(X_{N(w)}=0)$, the probability that no neighbour of $w$ is selected. Conditional on a particular selected subset of $U$, a vertex $w$ with a selected neighbour is forbidden, and all other vertices of $W$ are available. Because there are no edges within $W$, every subset of the available vertices is a valid completion. Uniform choice of $I$ makes these completions equally likely. The available coordinates are therefore independent fair binary choices, each of entropy one, and the forbidden coordinates are zero. Averaging their number givesBy the chain rule for information entropy,It remains to bound each bracket. Let $Y=X_{N(w)}$, a binary vector of length $s$, with distribution $(p_y)$ on $\{0,1{\}}^s$, and put $Z=2^r+2^s-1$. Assign weight $w_0=2^r$ to the all-zero vector and weight $w_y=1$ to every nonzero vector. ThenThe first inequality is Jensen's inequality for the concave logarithm; the weights sum to $2^r+(2^s-1)$. This explicitly handles zero probabilities and does not assume the neighbourhood indicators are independent.

There are $n$ right vertices, so combining the bounds and using $rm=sn$ givesExponentiating proves the required independent-set bound for biregular graphs. The proof uses both degree constraints: $r$ supplies the neighbourhood-cover multiplicity and the conditional-completion weight, while $s$ determines the number of neighbourhood patterns.

Take information entropy in bits and put $H(X_{\varnothing })=0$. Write $C=A\cap B$, $U=A\setminus B$, and $V=B\setminus A$. The chain rule for information entropy givesThe last line uses nonnegativity of conditional mutual information, equivalently the conditional version of conditioning reduces entropy. All random variables are finite-valued, so every conditional entropy here is finite. Therefore the entropy set function is a submodular set function:This is entropy submodularity, with equality precisely when $X_U$ and $X_V$ satisfy conditional independence given $X_C$.

On a probability measure-preserving system, for a finite measurable partition $\xi =\{A_1,\dots ,A_r\}$, the entropy of a finite measurable partition isUse natural logarithms, so information entropy is measured in nats; another fixed logarithm base rescales all answers. The join of measurable partitions is their common refinement, and write ${\xi }_a^b={\bigvee }_{j=a}^b{T}^{-j}\xi$, with an empty join the trivial partition. The entropy rate of a measurable partition and Kolmogorov-Sinai entropy are respectivelyThe block entropies form a subadditive sequence, so the first limit exists by the Fekete lemma. For finite partitions the conditional entropy of finite measurable partitions is $H(\eta \mid \zeta )=H(\eta \vee \zeta )-H(\zeta )$.

Put $c_0=H(\xi )$ and $c_k=H(\xi \mid {\xi }_1^k)$ for $k\ge 1$. The chain rule for information entropy, applied from the last coordinate backwards, and measure preservation giveThe second equality uses invariance of the joint partition atom probabilities under the common pullback $T^{-j}$; invertibility is unnecessary. Since conditioning reduces entropy, $c_k$ decreases to a nonnegative limit $c$. The Cesaro convergence of a sequence of this convergent sequence has the same limit. ThereforeEquivalently $h_{\mu }(T,\xi )=H_{\mu }(\xi \mid {\mathcal{F}}_1)$, where ${\mathcal{F}}_1=\sigma ({\bigvee }_{j\ge 1}{T}^{-j}\xi )$. Here conditional entropy of a countable measurable partition conditioned on a sigma-algebra is computed using conditional partition atom probabilities; the martingale convergence theorem gives continuity under increasing conditioning sigma-algebras. This is the infinite-future formula for partition entropy rate.

The Kolmogorov-Sinai generator theorem states that if a finite or countable measurable partition $\eta$ has finite entropy of a countable measurable partition and its iterates generate the whole completed sigma-algebra modulo null sets, then $h_{\mu }(T)=h_{\mu }(T,\eta )$. For an invertible system, generating means $\sigma ({\bigvee }_{j\in \mathbb{Z}}{T}^{-j}\eta )=\mathcal{B}$ modulo null sets. For a noninvertible system a one-sided generator, using $j\ge 0$, suffices. The two-sided and one-sided versions must not be confused.

For a Bernoulli shift with discrete symbol probabilities $(p_i)$, the coordinate-zero measurable partition has independent coordinate iterates. ThusFor a finite alphabet the coordinate partition is a generator of finite entropy of a finite measurable partition; on the two-sided sequence space use all integer coordinate iterates, and on the one-sided space use the nonnegative ones. The Kolmogorov-Sinai generator theorem proves the displayed answer in both cases. The same calculation applies to countably many symbols when their Shannon entropy is finite, using the countable finite-entropy version of the theorem. If the Shannon entropy is infinite, merge all but the first $r$ symbols into one cell. These finite coordinate partitions have entropy rate $-{\sum }_{i\le r}{p}_i\log p_i-p_{>r}\log p_{>r}\to \infty$, so the system entropy is infinite. Zero-probability symbols contribute zero. In particular a fair $r$-symbol shift has entropy $\log r$.

For the final assertion, let ${\mathcal{F}}_0=\sigma ({\bigvee }_{j\ge 0}{T}^{-j}\xi )$ and complete it modulo null sets. The approximation property forces ${\mathcal{F}}_0=\mathcal{B}$ modulo null sets. Indeed for each $A\in \mathcal{B}$, choose approximating sets from finite blocks with error tending to zero. Their indicators approach $1_A$ in $L^2$, and $L^2({\mathcal{F}}_0)$ is a closed vector subspace, so $A$ is ${\mathcal{F}}_0$-measurable modulo a null set.

Invertibility now gives ${\mathcal{F}}_1=T^{-1}{\mathcal{F}}_0=T^{-1}\mathcal{B}=\mathcal{B}$ modulo null sets. In particular the present partition $\xi$ is measurable with respect to its entire future, so $H(\xi \mid {\mathcal{F}}_1)=0$. The infinite-future formula gives $h_{\mu }(T,\xi )=0$. Since the given one-sided generator is also a two-sided generator, the Kolmogorov-Sinai generator theorem finishes the proof:This is finite one-sided generator of an invertible system forces zero entropy. Invertibility is essential: a fair binary one-sided Bernoulli shift has a finite one-sided generator and Kolmogorov-Sinai entropy $\log 2$.

For a probability measure-preserving system, the hypothesis is completely positive entropy: every finite measurable partition with positive static entropy of a finite measurable partition has positive entropy rate of a measurable partition. Fix a finite partition $\xi$ and putAll sigma-algebras are interpreted modulo null sets. We will prove that every finite partition $\alpha$ measurable with respect to this tail sigma-algebra of a measurable partition has $h_{\mu }(T,\alpha )=0$. Applying this to a binary partition will force the required triviality. This proves the needed direction of the Tail characterization of the Pinsker sigma-algebra directly, including noninvertible transformations.

Write $h=h_{\mu }(T,\xi )$. The infinite-future entropy formula and the backwards chain rule for information entropy give, for every $L\ge 1$, the block conditional entropy given the infinite future identityEach term equals $h$ by invariance of the joint probabilities under a common pullback and continuity of conditional entropy under increasing finite future blocks. Invertibility is not needed for this identity.

Fix $\epsilon >0$. Since $\alpha$ is ${\mathcal{F}}_0$-measurable and finite, the martingale convergence theorem and continuity of finite-partition conditional entropy allow an $r\ge 0$ withTo see the continuity explicitly, for each partition atom $A$ of $\alpha$ the conditional probabilities $\mathbb{E}[1_A\mid \sigma ({\xi }_0^r)]$ tend to $1_A$ almost everywhere; apply dominated convergence theorem to the bounded continuous function $-t\log t$ on $[0,1]$ and sum over the finitely many partition atoms.

For $n\ge 1$ set $\gamma ={\alpha }_0^{n-1}$, $L=n+r$, and $\beta ={\xi }_0^{L-1}$. The conditional entropy of finite measurable partitions satisfiesThe second inequality uses conditioning reduces entropy, since $\beta$ refines each block $T^{-j}{\xi }_0^r$; the final equality uses measure preservation.

Moreover $\gamma$ is ${\mathcal{F}}_L$-measurable. For each $j\ge 0$, tail measurability gives $\alpha$ measurable with respect to ${\mathcal{F}}_L$, hence $T^{-j}\alpha$ measurable with respect to $T^{-j}{\mathcal{F}}_L={\mathcal{F}}_{L+j}\subseteq {\mathcal{F}}_L$. Thus conditioning reduces entropy and the displayed block identity implyUse the symmetric entropy identity $H(\gamma )=H(\beta )-H(\beta \mid \gamma )+H(\gamma \mid \beta )$ to obtainThe fraction on the right tends to zero: $r$ is fixed and $H({\xi }_0^{n+r-1})/(n+r)\to h$ by the definition of entropy rate of a measurable partition. Taking $n\to \infty$ and then $\epsilon \downarrow 0$ gives

Now let $A\in \mathcal{T}(\xi )$ and take $\alpha =\{A,X\setminus A\}$. If $0<\mu (A)<1$, its binary entropy is $-\mu (A)\log \mu (A)-(1-\mu (A))\log (1-\mu (A))>0$, while its entropy rate is zero, contradicting completely positive entropy. ThereforeSince $\xi$ was arbitrary, every finite-partition tail is trivial. The argument needs only that $\xi$ is finite and the measure is a probability; it does not assume a finite generator, finite total system entropy, or invertibility.

Let $Z$ indicate whether the outcome differs from $x_1$. Then $\mathbb{P}(Z=1)=q$. The chain rule for information entropy givessince $X=x_1$ is determined when $Z=0$. The remaining conditional distribution has at most $m-1$ outcomes, so the maximum entropy on a finite alphabet is ${\log }_2(m-1)$. Therefore the Shannon source coding theorem limit obeysHere $h_2$ is the binary entropy function; equality holds when the rare outcomes are equiprobable. A convenient explicit small-$q$ bound isbecause $-(1-q){\log }_2(1-q)\le q{\log }_{2}e$. For a fixed alphabet, this upper bound tends to zero as $q\to 0$.

For finite-valued discrete random variables, the conditional entropy isThe chain rule for information entropy in two orders givesThereforeusing nonnegativity of conditional entropy and the fact that conditioning reduces entropy. This proves the required inequality.

For Fano's inequality, let $X$ take values in an alphabet of size $m$, let $\hat{X}=g(Y)$ be any estimator, and putBecause $E$ is determined by $(X,Y)$,Now $H(E\mid Y)\le H(E)=h_2(p_e)$. If $E=0$, $X$ is determined by $Y$; if $E=1$, at most $m-1$ values remain possible. HenceCombining the bounds yields

Shearer's inequality states that if $X=(X_1,\dots ,X_n)$ is a discrete random vector and $\mathcal{F}$ is a collection of subsets of $[n]$ in which every index occurs at least $t$ times, then

Order the coordinates naturally. For every $F\in \mathcal{F}$, the chain rule for information entropy givesRemoving conditioning variables cannot decrease entropy, so every summand is at leastAfter summing over $F$, each index $i$ contributes at least $t$ times. A final application of the chain rule yieldswhich proves the lemma.

The chain rule for information entropy and conditioning reduces entropy giveThus subadditivity of information entropy holds for two finitely valued random variables. Applying this inequality to $(X_1,\dots ,X_{n-1})$ and $X_n$ and then inducting gives

For $S_i=[n]\setminus \{i\}$, the chain rule for information entropy givesEach summand is at least $H(X_j\mid X_1^{j-1})$ because conditioning reduces entropy. Summing over $i$, each $j$ occurs $n-1$ times:This is the required special case of Shearer's inequality.

The axioms for information entropy give the formula $H(X)=-{\sum }_{x}p(x)\log p(x)$ and hence the chain rule for information entropyBecause conditioning reduces entropy, $H(Y\mid X)\le H(Y)$, and thereforeThis is subadditivity of information entropy.

For the entropy submodularity rule, apply the chain rule for information entropy twice:The last quantity is conditional mutual information, whose nonnegativity again expresses that conditioning reduces entropy.

Write $r=|V_0|$, $s=|V_1|$, and $e=|E(G)|=\alpha |X||Y|$. We construct a random graph homomorphism $\Psi :T\to G$. Choose a root of the tree. Map it according to the degree-biased vertex distribution on the corresponding side of the bipartite graph; after mapping any vertex, map each child independently and uniformly to a neighbour of its parent's image.

Every oriented tree edge is then mapped uniformly onto the $e$ edges of $G$. Every tree vertex in $V_0$ has the degree-biased marginal on $X$, of entropy $H_X$, and every vertex in $V_1$ has the analogous marginal of entropy $H_Y$. Repeated use of the chain rule for information entropy along the rooted tree givesSince a $k$-vertex tree has $k-1$ edges,The maximum entropy distribution on a finite set gives $H_X\le \log |X|$ and $H_Y\le \log |Y|$. HenceIf $N$ is the number of bipartition-respecting graph homomorphisms $T\to G$, the support of $\Psi$ has size $N$, so the maximum entropy distribution on a finite set also gives $H(\Psi )\le \log N$. ThusThere are $|X{|}^r|Y{|}^s$ bipartition-respecting maps in total, so a uniformly chosen one is a graph homomorphism with probability at least ${\alpha }^{k-1}$. This proves the Sidorenko inequality for trees.

Let $\mathcal{Q}$ be the set of valid quintuples and choose $(A_1,\dots ,A_5)$ uniformly from $\mathcal{Q}$. ThenApply Shearer's inequality to the ten pairs $\{i,j\}\subseteq [5]$. Every index occurs in four pairs, so

Put $U_{ij}=A_i\cup A_j$. Since $U_{ij}\in \mathcal{A}$, it has cardinality $k$. For each element of $U_{ij}$, its membership bits in $(A_i,A_j)$ are one of $(1,0),(0,1),(1,1)$; outside $U_{ij}$ they are forced to be $(0,0)$. Thus at most $3^k$ ordered pairs have any prescribed union. The chain rule for information entropy, conditional entropy, and the support bound for information entropy givebecause every $U_{ij}$ belongs to $\mathcal{A}$. There are ten pairs, henceExponentiating proveswhich is the five-variable case of the entropy bound for pairwise-union tuples.

Letwhere $\phi$ is the golden ratio. If the union-closed family consists of one nonempty set, any element of that set has frequency one, so assume its cardinality exceeds one. Choose independent uniform members $A,B$ and let $X,Y\in \{0,1{\}}^n$ be their characteristic vectors of sets. Then $H(X)=H(Y)=\log |\mathcal{A}|>0$.

Suppose for a contradiction that every element has frequency $p_i<c$. Put $q_i=1-p_i>1/\phi$, and let $Z=X\mathrm{OR}Y$, the characteristic vector of a set of $A\cup B$. The chain rule for information entropy and conditioning reduces entropy givebecause $Z_{<i}$ is a function of $(X_{<i},Y_{<i})$.

Fix the two prefixes and setThe two conditioned bits are independent random variables, and $Z_i=0$ exactly when both are zero. The supplied binary entropy product inequality therefore givesAveraging over the independent prefixes yieldswhenever either conditional entropy is positive. Summing and using $H(X)>0$ gives $H(Z)>H(X)$.

But set union keeps $A\cup B$ inside the union-closed family, so $Z$ is supported on $\mathcal{A}$. The maximum entropy distribution on a finite set gives $H(Z)\le \log |\mathcal{A}|=H(X)$, a contradiction. Some element must therefore occur in at least $c|\mathcal{A}|$ members, proving the entropy bound for a union-closed family.

The Entropic Balog-Szemerédi-Gowers theorem states that for finitely supported random variables $A,B$ in an abelian group,where the left-hand side is the Simultaneous conditional entropic Ruzsa distance.

Put $S=A+B$ and take two copies $(A_1,B_1)$ and $(A_2,B_2)$ that are conditionally independent given $S$. Thus both sums equal $S$, and, given $S$, $A_1$ and $B_2$ are independent with the required conditional marginals. By conditioning reduces entropy,The two conditional entropies are equal, since either $A$ or $B$ together with $S$ determines the other, and the chain rule for information entropy gives

Set $D=A_1-B_2$. By entropy submodularity,The first term is $H(A_1,B_2)\le H(A)+H(B)$ by subadditivity of information entropy. Since $A_1+B_1=A_2+B_2$, we also have $D=A_2-B_1$, so the second term is at most $H(A)+H(B)$. Finally $(D,A_1,B_1)$ determines all four copied variables, and conditional independence givesAs $H(A,B)=H(A)+H(B)-I(A;B)$, these estimates implySubtracting the common conditional-entropy term proves the stated Entropic Balog-Szemerédi-Gowers theorem.

Because the Entropic Ruzsa distance depends only on marginal distributions, take $X,Y,Z$ independent with the required marginals. Since $X-Z=(X-Y)+(Y-Z)$ is a function of $(X-Y,Y-Z)$, the data processing inequality for mutual information yieldsThe map $(X,X-Y,Y-Z)\mapsto (X,Y,Z)$ is a bijection. Using independence and the chain rule for information entropy, the left side iswhereas the right side is $H(X-Z)-H(Z)$. Hencewhere the final step is subadditivity of information entropy. Substituting this inequality into the definition of $d_R$ gives the Entropic Ruzsa triangle inequality

Put $L=L^{\ast }(X)$ and $\mu =\mathbb{E}L$. Since $L$ is a function of $X$, the chain rule for information entropy and part (c) givePart (a), applied to the nonnegative integer-valued random variable $L$, yieldsThe logarithm inequality $\log (1+t)\le t\log e$ implies $\mu \log (1+1/\mu )\le \log e$. ThusPart (c) also gives $\mu \le H(X)$, so monotonicity of the logarithm lets us replace $\log (1+\mu )$ by $\log (H(X)+1)$. Rearranging proves

Let $X=(X_1,\dots ,X_n)$ be a discrete random vector, and let $\mathcal{S}$ be a collection of subsets of $[n]$ in which every index occurs at least $r$ times. ThenApply the chain rule for information entropy to each projection $X_S$ and use conditioning reduces entropy to compare every term with the corresponding conditional entropy in the full chain rule.

The chain rule for information entropy and conditioning reduces entropy givewith equality exactly when $X$ and $Y$ are independent.