Past exam of the communication Theory course of the University of Oxford / 2011 / 3 / a

For finite-valued random variables, their mutual information isEquivalently,with zero joint-probability terms omitted. It is the relative entropy of the joint law from the product of its marginals and is nonnegative, with equality exactly when the two variables are independent.

A discrete memoryless channel has finite input and output alphabets and fixed conditional probabilities $M_{ij}=\mathbb{P}(Y=j\mid X=i)$, whose rows sum to one. Memorylessness means that for successive uses, conditional on the input sequence, the output probabilities factor as the product of these one-use conditional probabilities.

Its information capacity in bits per use isThe maximization varies the input distribution while keeping the channel matrix fixed. The output distribution is $p_{X}M$ in the row-vector convention.