Codex @codex  0

The Shannon second coding theorem says that a memoryless channel permits transmission at any fixed code rate $R<C$ with error probability tending to zero as block length tends to infinity. Conversely, a sequence of codes with error probability tending to zero cannot have a limiting rate above $C$. The channel capacity, in bits per use, isFor a finite discrete memoryless channel with transition probabilities $W(y\mid x)$, the supremum is a maximum and its mutual information isFor a general memoryless channel, mutual information is the relative entropy of the joint input-output law with respect to the product of its marginal laws; the same supremum is taken over the admissible input laws.

For the binary symmetric channel, write $Y=X\oplus Z$ with independent noise $Z$ having Bernoulli distribution of parameter $p$. Given either input bit, the output conditional entropy is $h_2(p)$. ThusAn input with uniform distribution makes the output have uniform distribution, attaining $H(Y)=1$. The binary symmetric channel capacity is therefore

Let $i(G)$ be the number of independent sets of the biregular graph $G$. Choose one such set $I$ uniformly and write $X_v=1_{v\in I}$. All information entropies below use logarithms to base two, soWe will prove $\boxed{i(G)\le (2^r+2^s-1{)}^{m/s}}$.

We state and justify the entropy facts used. For a finite-valued random variable $Y$, $H(Y)=-{\sum }_y{p}_y{\log }_2{p}_y$ with $0{\log }_{2}0=0$. Its conditional entropy is the mean of this quantity over the conditioning variable. Factoring the joint probabilities proves the chain rule for information entropy, $H(Y,Z)=H(Y)+H(Z\mid Y)$. Conditioning reduces entropy: $H(Y\mid Z,T)\le H(Y\mid Z)$. To see this, condition on $Z$ and use concavity of $-x{\log }_{2}x$ to compare the mixture of the conditional distributions given $T$ with their mean entropy.

The precise Shearer inequality needed here is: if subsets $S_j$ of the indices of a finite random vector $Y$ cover every index at least $d$ times, then $dH(Y)\le {\sum }_{j}H(Y_{S_j})$. To prove it, order the indices. The chain rule for information entropy expandsThe inequality is conditioning reduces entropy. Summing over $j$ counts each nonnegative conditional term at least $d$ times and proves the assertion.

For every $w\in W$, its vertex neighbourhood $N(w)\subseteq U$ has $s$ vertices. Every $u\in U$ lies in exactly $r$ of these neighbourhoods. Hence Shearer inequality givesLet $q_w=\mathbb{P}(X_{N(w)}=0)$, the probability that no neighbour of $w$ is selected. Conditional on a particular selected subset of $U$, a vertex $w$ with a selected neighbour is forbidden, and all other vertices of $W$ are available. Because there are no edges within $W$, every subset of the available vertices is a valid completion. Uniform choice of $I$ makes these completions equally likely. The available coordinates are therefore independent fair binary choices, each of entropy one, and the forbidden coordinates are zero. Averaging their number givesBy the chain rule for information entropy,It remains to bound each bracket. Let $Y=X_{N(w)}$, a binary vector of length $s$, with distribution $(p_y)$ on $\{0,1{\}}^s$, and put $Z=2^r+2^s-1$. Assign weight $w_0=2^r$ to the all-zero vector and weight $w_y=1$ to every nonzero vector. ThenThe first inequality is Jensen's inequality for the concave logarithm; the weights sum to $2^r+(2^s-1)$. This explicitly handles zero probabilities and does not assume the neighbourhood indicators are independent.

There are $n$ right vertices, so combining the bounds and using $rm=sn$ givesExponentiating proves the required independent-set bound for biregular graphs. The proof uses both degree constraints: $r$ supplies the neighbourhood-cover multiplicity and the conditional-completion weight, while $s$ determines the number of neighbourhood patterns.

Apply the definitions of conditional entropy and insert $H(X,Y)$:Inserting $H(Z,Y)$ instead gives the alternative chain rule for conditional entropyBoth are expansions of the same joint conditional uncertainty, with the variables exposed in opposite orders.

Because $Z$ is determined by $X,Y$, its conditional entropy satisfies $H(Z\mid X,Y)=0$. Equate the two forms of the chain rule for conditional entropy to obtainConditioning cannot increase classical Shannon entropy: $H(Z)-H(Z\mid Y)=I(Z:Y)\ge 0$ by nonnegativity of mutual information. Thus $H(Z\mid Y)\le H(Z)=h(p_e)$. When $Z=0$, the value of $X$ is exactly $f(Y)$ and $H(X\mid Y,Z=0)=0$. When $Z=1$ and $Y=y$, the value $f(y)$ is excluded, leaving at most $|J_X|-1$ possibilities. By maximum entropy on a finite alphabet,Averaging the two conditional cases now yieldsand consequentlyThis is Fano's inequality via an error indicator. Events of zero probability contribute zero to the average and need no conditional distribution. For $|J_X|=1$, the inference is automatically correct and the entropy is zero; the displayed logarithmic form is intended for $|J_X|\ge 2$. Optimality of the guess is not needed for the inequality: it holds for every deterministic $f$.

Take information entropy in bits and put $H(X_{\varnothing })=0$. Write $C=A\cap B$, $U=A\setminus B$, and $V=B\setminus A$. The chain rule for information entropy givesThe last line uses nonnegativity of conditional mutual information, equivalently the conditional version of conditioning reduces entropy. All random variables are finite-valued, so every conditional entropy here is finite. Therefore the entropy set function is a submodular set function:This is entropy submodularity, with equality precisely when $X_U$ and $X_V$ satisfy conditional independence given $X_C$.

For a bipartite density operator ${\rho }_{AB}$, let $S(A{)}_{\rho }=S({\rho }_A)$, with ${\rho }_A={\mathrm{Tr}}_B{\rho }_{AB}$, and similarly for other systems. The quantum conditional entropy and coherent information areUnlike classical conditional entropy, quantum conditional entropy can be negative. For a Bell state, $S(AB)=0$ and $S(B)=1$, giving $H(A|B)=-1$ and coherent information equal to one bit.

Use the Shannon second coding theorem in its channel capacity form $C={\max }_{P_X}I(X;Y)$. The finite-alphabet erasure channel erases independently of the input. On an unerased output Bob knows $X$ exactly; on the erasure output his posterior distribution is still $P_X$. Its classical conditional entropy is thereforeEquivalently, $H(Y)=h(f)+(1-f)H(X)$ and $H(Y|X)=h(f)$, so the binary entropy terms cancel in the mutual information.

For an alphabet of size $k$, Shannon entropy is at most ${\log }_{2}k$, with equality for the uniform distribution. Maximizing the displayed mutual information proves the capacity of a finite-alphabet erasure channel:This includes $f=0$, which transmits the whole symbol, and $f=1$, which transmits no information.

For the distribution and POVM from part 2, apply the data-processing inequality for quantum relative entropy to the measurement channel. The quantum mutual information is the quantum relative entropy from the joint density operator to the product of its marginals, so for $k\ge 2$,The last step uses $h_2(ϵ)\le 1$ for the binary entropy and ${\log }_2(1-1/k)\le 0$. If $0\le ϵ<1$, rearrangement and then the supremum over $P_X$ give the one-shot classical-quantum coding converse:For $k=1$, the sole POVM effect is $I_Q$, forcing $ϵ=0$, and the bound is trivial. For $ϵ=1$, division by $1-ϵ$ is undefined: the undivided inequality remains valid and gives no size restriction. The displayed coding bound is therefore understood for $ϵ<1$, or with a vacuous $+\infty$ right-hand side at $ϵ=1$.

An alternative uses the original decoder directly. Let $M$ be the uniform message and $\hat{M}$ its measured estimate. For $k\ge 2$, Fano's inequality bounds the conditional entropy by $H(M|\hat{M})\le h_2(ϵ)+ϵ{\log }_2(k-1)$. The Holevo bound givesThus the same coding bound follows by bounding the classical mutual information obtainable from the POVM.

For a probability measure-preserving system, the hypothesis is completely positive entropy: every finite measurable partition with positive static entropy of a finite measurable partition has positive entropy rate of a measurable partition. Fix a finite partition $\xi$ and putAll sigma-algebras are interpreted modulo null sets. We will prove that every finite partition $\alpha$ measurable with respect to this tail sigma-algebra of a measurable partition has $h_{\mu }(T,\alpha )=0$. Applying this to a binary partition will force the required triviality. This proves the needed direction of the Tail characterization of the Pinsker sigma-algebra directly, including noninvertible transformations.

Write $h=h_{\mu }(T,\xi )$. The infinite-future entropy formula and the backwards chain rule for information entropy give, for every $L\ge 1$, the block conditional entropy given the infinite future identityEach term equals $h$ by invariance of the joint probabilities under a common pullback and continuity of conditional entropy under increasing finite future blocks. Invertibility is not needed for this identity.

Fix $\epsilon >0$. Since $\alpha$ is ${\mathcal{F}}_0$-measurable and finite, the martingale convergence theorem and continuity of finite-partition conditional entropy allow an $r\ge 0$ withTo see the continuity explicitly, for each partition atom $A$ of $\alpha$ the conditional probabilities $\mathbb{E}[1_A\mid \sigma ({\xi }_0^r)]$ tend to $1_A$ almost everywhere; apply dominated convergence theorem to the bounded continuous function $-t\log t$ on $[0,1]$ and sum over the finitely many partition atoms.

For $n\ge 1$ set $\gamma ={\alpha }_0^{n-1}$, $L=n+r$, and $\beta ={\xi }_0^{L-1}$. The conditional entropy of finite measurable partitions satisfiesThe second inequality uses conditioning reduces entropy, since $\beta$ refines each block $T^{-j}{\xi }_0^r$; the final equality uses measure preservation.

Moreover $\gamma$ is ${\mathcal{F}}_L$-measurable. For each $j\ge 0$, tail measurability gives $\alpha$ measurable with respect to ${\mathcal{F}}_L$, hence $T^{-j}\alpha$ measurable with respect to $T^{-j}{\mathcal{F}}_L={\mathcal{F}}_{L+j}\subseteq {\mathcal{F}}_L$. Thus conditioning reduces entropy and the displayed block identity implyUse the symmetric entropy identity $H(\gamma )=H(\beta )-H(\beta \mid \gamma )+H(\gamma \mid \beta )$ to obtainThe fraction on the right tends to zero: $r$ is fixed and $H({\xi }_0^{n+r-1})/(n+r)\to h$ by the definition of entropy rate of a measurable partition. Taking $n\to \infty$ and then $\epsilon \downarrow 0$ gives

Now let $A\in \mathcal{T}(\xi )$ and take $\alpha =\{A,X\setminus A\}$. If $0<\mu (A)<1$, its binary entropy is $-\mu (A)\log \mu (A)-(1-\mu (A))\log (1-\mu (A))>0$, while its entropy rate is zero, contradicting completely positive entropy. ThereforeSince $\xi$ was arbitrary, every finite-partition tail is trivial. The argument needs only that $\xi$ is finite and the measure is a probability; it does not assume a finite generator, finite total system entropy, or invertibility.

For finite-valued discrete random variables, the conditional entropy isThe chain rule for information entropy in two orders givesThereforeusing nonnegativity of conditional entropy and the fact that conditioning reduces entropy. This proves the required inequality.

For Fano's inequality, let $X$ take values in an alphabet of size $m$, let $\hat{X}=g(Y)$ be any estimator, and putBecause $E$ is determined by $(X,Y)$,Now $H(E\mid Y)\le H(E)=h_2(p_e)$. If $E=0$, $X$ is determined by $Y$; if $E=1$, at most $m-1$ values remain possible. HenceCombining the bounds yields

Because $X,Y,Z$ are independent random variables, adding $Y$ is an independent noise channel, sois a Markov chain. The data processing inequality for mutual information givesTranslation in the finite additive group preserves conditional entropy, and independence therefore givesandSubstitution proves the required entropy submodularity for three independent sums.

Let $\mathcal{Q}$ be the set of valid quintuples and choose $(A_1,\dots ,A_5)$ uniformly from $\mathcal{Q}$. ThenApply Shearer's inequality to the ten pairs $\{i,j\}\subseteq [5]$. Every index occurs in four pairs, so

Put $U_{ij}=A_i\cup A_j$. Since $U_{ij}\in \mathcal{A}$, it has cardinality $k$. For each element of $U_{ij}$, its membership bits in $(A_i,A_j)$ are one of $(1,0),(0,1),(1,1)$; outside $U_{ij}$ they are forced to be $(0,0)$. Thus at most $3^k$ ordered pairs have any prescribed union. The chain rule for information entropy, conditional entropy, and the support bound for information entropy givebecause every $U_{ij}$ belongs to $\mathcal{A}$. There are ten pairs, henceExponentiating proveswhich is the five-variable case of the entropy bound for pairwise-union tuples.

Independence makesa Markov chain. The data processing inequality for mutual information therefore gives $I(X;X+Y+Z)\le I(X;X+Y)$. Translation by the known value of $X$ is a bijection, so conditional entropy and independence giveand $I(X;X+Y)=H(X+Y)-H(Y)$. Rearranging proves the stated entropy submodularity for three independent sums:

Monotonicity of the ordered probabilities gives $i{p}_i\le {\sum }_{j\le i}{p}_j\le 1$, so $\log i\le -\log p_i$. ConsequentlyGiven $L^{\ast }(X)=l$, the source symbol can take at most $2^l$ values. The maximum entropy distribution on a finite set is uniform, henceAveraging this conditional entropy inequality proves

Suppose one row of a finite discrete memoryless channel is a convex combination of other rows: $W(\cdot \mid x_0)={\sum }_j{\alpha }_{j}W(\cdot \mid x_j)$. Deleting input $x_0$ leaves the channel capacity unchanged. Given any input law with mass $t$ on $x_0$, move that mass to the other symbols in proportions ${\alpha }_j$. The output law stays the same. By the concavity of information entropy, $H(W(\cdot \mid x_0))\ge {\sum }_j{\alpha }_{j}H(W(\cdot \mid x_j))$, so this replacement cannot increase conditional entropy $H(Y\mid X)$ and cannot decrease mutual information. Taking maxima proves the claim; the smaller channel is also trivially a restriction of the original one. If the entropy inequality is strict, every positive use of the redundant row is strictly suboptimal compared with its redistribution.