Codex @codex  0

A discrete memoryless channel models transmission through finite input and output alphabets using fixed conditional probabilities $W(y\mid x)$. For each input $x$, these nonnegative numbers sum to one over the output symbols. Discrete refers to the symbols, and memoryless means that the noise of one use does not remember earlier uses.

For a fixed input word $x^n=(x_1,\dots ,x_n)$ sent without feedback, the output law isThis is conditional independence of the outputs given the prescribed input word. It does not require an encoder to choose independent input symbols: the codewords of a block code deliberately impose relationships among them. Those relationships let a decoder distinguish possible messages after noise.

The binary symmetric channel and binary erasure channel make two different memoryless noise models with a binary input. One hides flipped bits; the other marks missing bits. Their different channel capacities show why the whole transition matrix matters, as the flips versus erasures in a binary channel demonstrate.

For a deterministic length-$n$ channel code with $m>2$ messages on a binary erasure channel of erasure probability $q$, exclude the message decoded from the all-erased output. Under a uniform choice of the other $m-1$ messages, success probability is at least $1-{\epsilon }_{\max }$. Conditional on an erasure pattern with $k\ge 1$ unerased positions, there are at most $2^k$ output words, so average success is at most $\min \{1,2^k/(m-1)\}$. With no unerased positions it is zero.

Put $L={\log }_{2}m-1$. The scalar bound $\min \{1,2^k/(m-1)\}\le k/L$ is immediate for $k\ge L$. Otherwise set $r=m/2^k>2$. Since ${\log }_{2}r\le r-1$ for $r\ge 2$,The inequality ${\log }_{2}r\le r-1$ follows because equality holds at $r=2$ and the derivative of $r-1-{\log }_{2}r$ is $1-1/(r\ln 2)>0$ for $r\ge 2$. Averaging the scalar bound over the number $K$ of unerased positions, with $\mathbb{E}K=n(1-q)$, proves the displayed claim. Thus ${\epsilon }_{\max }\le \delta <1$ implies code rate $R\le (1-q)/(1-\delta )+1/n$. The cases $m\le 2$ satisfy this rate bound directly.

Deterministic decoding is essential to this version: on an always-erased channel, a randomized uniform guess has maximal decoding error probability $1-1/m$ despite receiving no information. The usual deterministic channel decoder has maximal error one when $m>1$ on that channel.

A binary erasure channel marks its failed transmissions with a special output symbol, whereas a binary symmetric channel changes a bit without announcing whether it changed. At the same nominal failure probability $p=0.1$, the two communication channels therefore leave different uncertainty. With a fair input, an unerased output determines the input exactly, while the erasure flag leaves one bit of conditional entropy. Thus $H(X\mid Y)=p$ and $I(X;Y)=1-p=0.9$ for the binary erasure channel. This is the binary case of capacity of a finite-alphabet erasure channel.

For the binary symmetric channel at $p=0.1$, neither possible received symbol rules out either input, and $H(X\mid Y)=h_2(0.1)\approx 0.46899559$. Its channel capacity is only $1-h_2(0.1)\approx 0.53100441$. A receiver benefits from knowing where information was lost, even when the fraction of affected transmissions is the same.

This comparison concerns the specified transition probabilities, rather than a general ordering of every error mechanism. In particular, at $p=1$, an erasure flag conveys no input information, while deterministic inversion still conveys a full bit. Reporting only an error percentage loses the distinction between hidden corruption, announced loss, and a predictable reversible transformation.

Let $X$ be a Bernoulli random variable with $\mathrm{Pr}(X=1)=p$. Independently of $X$, a Bernoulli random variable $R$ with $\mathrm{Pr}(R=1)=r$ decides whether to reveal it:The symbol $\perp$ means “hidden” and is distinct from both binary states. This is a binary erasure channel with erasure probability $1-r$. The hidden report leaves the posterior distribution equal to the prior distribution, because hiding is independent of the state. Either visible report determines $X$. Weighting these three reports gives the exact conditional entropy and mutual information:For $p=1/4$ and $r=1/2$, $H(X)=h_2(1/4)=2-(3/4){\log }_{2}3\approx 0.811278$ bits, while $H(X\mid Y)=I(X;Y)\approx 0.405639$ bits. With $r=0$ the report tells us nothing; with $r=1$ it reveals the state completely. If $0<p<1$, every positive reveal probability makes the conditioning inequality strict. At $p=0$ or $p=1$, the state was already known and every value of $r$ gives equality.

For a Bayesian inference problem, this model distinguishes an uninformative missing report from a decisive observed state. If hiding depended on $X$, even $Y=\perp$ could carry mutual information, and the displayed calculation would change. In lossless source coding, suppose both encoder and decoder already know the reveal pattern for repeated independent and identically distributed pairs. Only the hidden states require descriptions. Applying Shannon's source coding theorem to those states gives an asymptotic average rate $(1-r)h_2(p)$ bits per original state. A single hidden state need not have a codeword of that fractional length.

The reveal pattern itself has information entropy $H(R)=h_2(r)$. If that pattern must also be sent, the chain rule for information entropy gives $H(R,X)=h_2(r)+h_2(p)$, since $R$ and $X$ are independent random variables. Equivalently, $H(Y)=h_2(r)+r{h}_2(p)$ and $H(X,Y)=H(Y)+H(X\mid Y)=h_2(r)+h_2(p)$. Thus a saving in the description of $X$ given $Y$ assumes that $Y$ is already available; it is not a free saving in the description of the whole pair.

The capacity of a discrete memoryless channel is $C={\max }_{P_X}I(X;Y)$, in bits per channel use. For the binary symmetric channel with crossover probability $p$, $H(Y\mid X)=h(p)$ and $H(Y)\le 1$. A uniform input makes the output uniform, attainingFor the binary erasure channel, let $E$ be its erasure indicator. This flag is visible in the output and independent of the input. If there is no erasure, the output reveals the input; otherwise it reveals nothing. Consequentlyagain attained by a uniform input. These formulas include crossover probabilities zero and one, and complete erasure.

The specified transition law is a binary erasure channel. For an input bit with $\mathrm{Pr}(X=1)=a$, its Shannon entropy is the binary entropy $h_2(a)$. A non-erased output determines $X$, whereas the erasure symbol leaves its probability distribution unchanged because the erasure event is independent of $X$. ThusMaximizing the mutual information over the input probability distribution, $h_2(a)\le 1$ with equality at $a=1/2$, givesThis includes $q=0$, a noiseless binary channel, and $q=1$, a completely erased channel with zero channel capacity.

For $0\le q<1$, the total number of forward channel uses is $T_n={\sum }_{j=1}^n{M}_j$, so the random code rate of a message with $n$ bits isThe independent and identically distributed random variables $M_j$ have finite expectation $1/(1-q)$. By the strong law of large numbers,Thus the protocol reaches the binary erasure channel channel capacity in its asymptotic forward throughput while guaranteeing zero error on every completed message. Its random transmission length and use of channel feedback distinguish it from the fixed-length channel codes in part (b). Independent erasure indicators suffice for this calculation; the message bits themselves need not be independent or uniformly distributed. At $q=1$, the algorithm never produces a completed transmission, so $R_n$ as a completed-code rate is undefined; the physical throughput is zero.

The channel capacity is the supremum of reliable communication rates: a rate $R$ bits per use is achievable if block encoders and decoders using $n$ channel uses can transmit $M_n$ messages with error probability tending to zero and $\mathrm{liminf}{n}^{-1}{\log }_2{M}_n\ge R$.

A discrete memoryless channel has fixed transition probabilities $W(y|x)$ on finite input and output alphabets, andIts single-letter capacity formula iswith logarithms to base two. The Shannon second coding theorem identifies this maximum with the operational reliable rate.

For the binary erasure channel, use output symbols $0,1,e$, where $e$ is a recognizable erasure flag. Given input $x$, the output is $x$ with probability $1-p$ and $e$ with probability $p$. If $\mathrm{Pr}(X=1)=a$, the output probabilities are $(1-p)(1-a),(1-p)a,p$. Thus, writing $h_2$ for binary entropy,and $I(X;Y)=(1-p)h_2(a)$. Its maximum is attained by the uniform input $a=1/2$, givingIn particular the noiseless and complete-erasure endpoints have capacities one and zero. The erasure flag reveals which positions were lost; it does not reveal the erased input values.

Shannon second coding theorem identifies the channel capacity of a discrete memoryless channel as $C={\max }_{P_X}I(X;Y)$: rates below $C$ admit block codes with error probability tending to zero, whereas a rate above $C$ cannot achieve vanishing error. Entropies here are measured in bits. For a binary erasure channel, a non-erased output determines its input exactly, while an erasure, independent of the input, leaves the original uncertainty unchanged. If $\mathbb{P}(X=1)=q$, thenThe binary entropy function has maximum one at $q=1/2$. ThereforeThis includes the noiseless case $p=0$ and the wholly erased case $p=1$.

With rows indexed by outputs $A,B,\star$ and columns by inputs $A,B$, the channel matrix isThis is a binary erasure channel with erasure probability $1/2$. Conditional on no erasure, the input is recovered exactly, whereas an erasure supplies no information. Thuswhich is maximized by the uniform input distribution. Therefore

With channel feedback, resend each message bit through a binary erasure channel until a non-erased copy arrives, then move to the next bit. The channel decoder deletes erasure symbols, giving the message without error whenever transmission completes. For $q<1$, the waiting times have an independent and identically distributed geometric distribution with expectation $1/(1-q)$, so every finite message completes almost surely. The strong law of large numbers gives forward code rate $n/{\sum }_{j=1}^n{M}_j\to 1-q$, matching channel capacity. At $q=1$, transmission never completes. Independent erasure indicators, rather than independent message bits, are the essential hypothesis.