Past exam of the communication Theory course of the University of Oxford / 2011 / 3 / b

Let $B$ be the selected block, and write $\alpha =\mathbb{P}(B=1)$. Because both the input and output alphabets are disjoint across blocks, $B$ is determined by either the input or the output. The mutual-information chain rule therefore givesFor any fixed $\alpha$, equality is achievable by using a capacity-achieving distribution inside each block. It remains to maximize this binary expression.

Put $S=2^{C_1}+2^{C_2}$ and $r=2^{C_1}/S$. A direct expansion of binary relative entropy yieldsPart 1(a) makes the maximum ${\log }_{2}S$, uniquely at $\alpha =r$. HenceThe block choice itself communicates information, in addition to any information sent within the selected block.

This is capacity of a disjoint sum of channels.

There are three distinguishable output groups: columns $\{1,2\}$ can arise only from input one; column $\{3\}$ can arise from inputs two or three, whose rows are identical; and columns $\{4,5\}$ can arise only from input four.

Each group has zero internal channel capacity. A block with one input cannot carry a variable input choice, and the middle block has identical output laws for both of its inputs. Combining the three disjoint zero-capacity blocks therefore givesEquivalently, the output identifies exactly which of these three groups was used, so mutual information is the entropy of that group label. It is maximized when all group probabilities are one third. The capacity-achieving input distributions are preciselySplitting the middle probability arbitrarily cannot communicate an extra distinction because its two channel rows agree.