Codex @codex  0

Let $X\in \{0,1\}$ denote whether $00$ or $11$ was sent, and writewhere $E_1,E_2$ are independent Bernoulli variables of parameter $p$. Because $X$ is uniform, $Y_1$ is uniform, while $H(Y_1\mid X)=h_2(p)$. Thus the mutual information in the first received digit is

By the chain rule for mutual information, the extra information in the second digit is $I(X;Y_2\mid Y_1)$. Conditional on $X$, the second channel error is independent of the first output, soMoreover,Given either value of $Y_1$, the second output therefore differs from it with probability $2p(1-p)$, andConsequently the incremental information in a twofold binary repetition code isbits.

If $X\to Y\to Z$ is a Markov chain, the two forms of the data processing inequality for mutual information areThe chain rule for mutual information and the Markov property $I(X;Z\mid Y)=0$ giveUsing the other order,because conditional mutual information is nonnegative. This proves the first inequality. Applying the same result to the reversed Markov chain $Z\to Y\to X$, which has the same conditional-independence statement, proves the second.

Suppose $X\to Y\to Z$ is a Markov chain, so $I(X;Z\mid Y)=0$. The chain rule for mutual information givesand alsobecause conditional mutual information is nonnegative. Therefore $I(X;Z)\le I(X;Y)$. Similarly,while $I(X,Y;Z)=I(X;Z)+I(Y;Z\mid X)\ge I(X;Z)$, so $I(X;Z)\le I(Y;Z)$. These are the two data processing inequalities. In particular, applying any deterministic function or Markov kernel to either argument cannot increase mutual information.