Input distribution optimization for a binary symmetric channel

ID: input-distribution-optimization-for-a-binary-symmetric-channel

Let for the input bit of a binary symmetric channel with crossover probability . The law of total probability gives
Consequently its mutual information is
Here specifies the noise and specifies the input bias; they are different choices. At and , the output has , giving and bits. With the same noise and the fair input , the output becomes fair and bits, attaining binary symmetric channel capacity.
Figure 1.
A fair input maximizes retained information through a binary symmetric channel
. The left panel holds the crossover probability fixed and varies . The right panel shows the output probability distributions for with input biases and . More output Shannon entropy helps here because the conditional noise information entropy stays fixed. Details.
For , binary entropy reaches one only at output mass , and the equation forces . Thus the maximizing input is unique in those cases. For , every input produces the same fair output and zero mutual information, so every input bias is optimal and none transmits information. A fixed input or also carries zero mutual information for every : noise can make its output unpredictable without creating an unknown message to recover.

New to topics? Read the docs here!