Past exam of the information theory course of the University of Oxford / 2021 / 3

For a discrete memoryless channel with transition probabilities $M(y\mid x)$, the conditional probability of a length-$n$ output given a length-$n$ input factors asThe same transition probabilities apply at each use. Its channel capacity isThe unit is bits per channel use. For finite alphabets the maximum exists by continuity on the probability simplex; with more general alphabets one uses a supremum when appropriate.

An $(m,n)$ channel code consists of an encoder $c:{\mathcal{V}}_m\to {\mathcal{X}}^n$ and a decoder $d:{\mathcal{Y}}^n\to {\mathcal{V}}_m$. It represents each of $m$ messages by $n$ input symbols and recovers a message from the $n$ observed output symbols. Its code rate is $R=({\log }_{2}m)/n$ bits per use. For each message $w$, its conditional error probability is $\mathbb{P}(d(Y^n)\ne w\mid W=w)$; an average error also specifies the message probability distribution, while the maximum error takes the largest conditional error.

Write $T=g(Y)$. Because $T$ is determined by $Y$, the chain rule for mutual information givesThis is the data processing inequality. It holds for every input probability distribution. Maximizing each side over that same set of input probability distributions gives $C_{g(Y)}\le C_Y$; processing cannot increase channel capacity.

A simple sufficient condition for equality is that $g$ be injective on the reachable output alphabet. Then $Y$ can be recovered from $T$, giving $H(Y\mid T)=0$ and hence $I(X;Y\mid T)=0$ for every input law. The input that achieves the original channel capacity therefore achieves it after processing as well. Values of $g$ outside reachable outputs are irrelevant.

Injectivity is sufficient rather than necessary. More precisely, for finite alphabets equality in capacity data processing holds exactly when there is an original capacity-achieving input law for which $X$ and $Y$ have conditional independence given $T$. Sufficiency follows from the displayed gap identity. For necessity, maximize $I(X;T)$ at an input law $p_{\ast }$; if its maximum equals $C_Y$, then $C_Y=I_{p_{\ast }}(X;T)\le I_{p_{\ast }}(X;Y)\le C_Y$, so both inequalities are equalities. Processing may discard output details unrelated to the input without losing channel capacity.

Interpret additive noise in the standard channel sense: $Z$ is independent of the chosen input $X$ and its law is fixed as the input law is varied. A uniform marginal law for $Z$ alone, without this independence assumption, would not specify the transition matrix. Only reachable output symbols are counted.

Translate the output alphabet by $-\min (z_1,z_2)$ and put $d=|z_2-z_1|\ge 1$. The reachable outputs becomeFor $d=1$ the two sets overlap in two points; for $d=2$ in one; for $d\ge 3$ they are disjoint. In particular $|\mathcal{Y}|\ge 4$, and choosing noise values $0,1$ attains four.

With rows indexed by inputs $0,1,2$ and columns by outputs $0,1,2,3$, independence and uniform two-point noise giveEvery conditional output distribution has information entropy one bit, so $I(X;Y)=H(Y)-1\le {\log }_{2}4-1=1$. The input probability mass function $(1/2,0,1/2)$ makes all four outputs equiprobable, attaining that bound. ThusUsing a zero-probability input in a channel-capacity maximization is legitimate; it is unrelated to the strictly positive source weights needed for the tree-shape claims in Question 2.

Each of the three input values has two possible outputs, so their union has at most six elements. The preceding union formula also gives this bound. Choose noise values $0,3$, whose two translated input alphabets are disjoint. With columns ordered $0,1,2,3,4,5$ the transition matrix isThus $|\mathcal{Y}|=6$. Every output determines its input, explicitly $X=Y\mathrm{mod}3$, so $H(X\mid Y)=0$. Consequently $I(X;Y)=H(X)\le {\log }_{2}3$, with equality for a uniform input. ThereforeThe extra uncertainty of which noise value occurred does not reduce information about the input: the two possible outputs for each input remain distinguishable from those of every other input.

Choose noise values $0,2$, giving exactly the outputs $0,1,2,3,4$. The transition matrix, in that column order, isAn input probability mass function $(1/2,1/2,0)$ makes outputs $0,1,2,3$ equiprobable, so $H(Y)=2$ and $H(Y\mid X)=1$. Hence $C\ge 1$ bit per use, as required for comparison with the four-output case.

There is a strict improvement, which can also be computed exactly. Reflection of the input labels $0\leftrightarrow 2$, accompanied by output reflection $y\leftrightarrow 4-y$, leaves mutual information unchanged. Concavity of information entropy makes mutual information a concave function of the input probability mass function for a fixed transition matrix. Averaging an input law with its reflection therefore cannot decrease it. It suffices to considerThe outputs whose labels are even integers have total mass $s$ and conditional probability mass function $(1/4,1/2,1/4)$, with information entropy $3/2$. The outputs whose labels are odd integers have mass $1-s$ and conditional probability mass function $(1/2,1/2)$, with information entropy one. Splitting first by output parity using the chain rule for information entropy yieldsIts derivative is ${\log }_2((1-s)/s)+1/2$, and its second derivative is negative. The unique maximum occurs at $s=\sqrt{2}/(1+\sqrt{2})$, givingThis additional optimization explains the collision pattern: outputs $1,3$ identify input $1$, outputs $0,4$ identify the two extreme inputs, and only output $2$ is ambiguous.

For every input probability mass function, conditional entropy is nonnegative, soTaking the maximum over input laws proves $C\le 1$ bit per channel use. This bound comes from the binary input alphabet itself; no specific transition probabilities or decoder are needed.

There are two possible messages and three channel uses, givingConditional on either transmitted message, a decoding error occurs exactly when at least two of the three bits are flipped. The discrete memoryless channel makes these flips independent, each with probability $\epsilon =1/10$. Their count therefore has a binomial distribution with parameters $3,\epsilon$. Majority-vote decoding of a repetition code givesInterchanging the two bit values leaves the binary symmetric channel and majority decision unchanged, so $\mathbb{P}(\text{error}\mid W=b)=7/250$ as well. Both the maximum conditional error and the average error, for any message prior, are therefore $0.028$. The probability of correct decoding for each message is $243/250=0.972$. These are finite-block error probabilities; a channel-capacity statement alone would not determine them.