Past exam of the communication Theory course of the University of Oxford / 2008 / 1

Write $p_{xy}=\mathbb{P}(X=x,Y=y)$, $p_x={\sum }_y{p}_{xy}$, and $q_y={\sum }_x{p}_{xy}$. In bits the mutual information isTerms with zero joint probability contribute zero. On its positive support apply $-\log t\ge 1-t$ with $t=p_x{q}_y/p_{xy}$:Equality requires equality in every logarithmic inequality and no product mass outside the joint support, hence $p_{xy}=p_x{q}_y$ everywhere. Conversely that factorization makes every term zero. Thus mutual information is nonnegative and vanishes exactly for independence. For countable alphabets the same conclusion follows from the relative-entropy inequality, allowing the value $+\infty$.

The entropy chain rule givesConditional entropy is nonnegative, so $I(X;Y)\le \min (H(X),H(Y))$. For finite entropies, if $H(X)\le H(Y)$, equality holds exactly when $H(X\mid Y)=0$, meaning $X$ is a function of $Y$ almost surely. If $H(Y)\le H(X)$, interchange the variables. When their entropies are equal, equality requires both functional relations. Zero conditional entropy has this characterization because each conditional probability distribution must be a point mass.

The conditional output distributions are permutations of the same triple, so both conditional entropies equalWith input probability $p$, the three output probabilities are $q_1=(2-p)/6$, $q_2=(1+2p)/6$, and $q_3=(3-p)/6$. ThereforeDifferentiating, the constant terms cancel because $\sum q_j^{\prime }=0$:Thus there is a unique maximum. Its stationary equation reduces to $3p^2+9p-5=0$, whose root in $[0,1]$ isThe other root is negative; strict concavity proves this is the global maximum.