= Conditional mutual information
{title2=$I(X;Y\mid Z)$}
{wiki}
For finite <discrete random variables>, conditional mutual information measures the average reduction in uncertainty about $X$ from observing $Y$, when $Z$ is already known. With base-two <logarithms>, it is measured in <bits>:
$$
I(X;Y\mid Z)=H(X\mid Z)-H(X\mid Y,Z)
=H(X\mid Z)+H(Y\mid Z)-H(X,Y\mid Z).
$$
The second expression follows from the <chain rule for conditional entropy> and makes the symmetry in $X,Y$ visible. Equivalently, compute the <mutual information> separately in each conditional <probability distribution> and then average:
$$
\begin{aligned}
I(X;Y\mid Z)
&=\sum_{z:p_Z(z)>0}p_Z(z)I(X;Y\mid Z=z)\\
&=\sum_{z:p_Z(z)>0}p_Z(z)
D\!\left(p_{X,Y\mid z}\,\middle\Vert\,p_{X\mid z}p_{Y\mid z}\right)\\
&=\sum_{x,y,z:p(x,y,z)>0}p(x,y,z)
\log_2\frac{p(x,y\mid z)}{p(x\mid z)p(y\mid z)}.
\end{aligned}
$$
Here $D$ is the <Kullback-Leibler divergence>, also computed with base-two <logarithms>; zero-probability terms contribute zero. The <Gibbs inequality> gives $I(X;Y\mid Z)\geq0$, with equality exactly when $X$ and $Y$ have <conditional independence> given $Z$: the conditional joint <probability mass function> factors for every $z$ of positive <probability>.<6.441 Information Theory (Spring 2016 lecture notes) by Yury Polyanskiy and others/Chapter 2: Information Measures: Mutual Information>[Definition 2.4 and Theorem 2.5(1)]{ref}
For a fully computed example, take <independent random variables> $X,Y$ that are fair <bits>, and let $Z=X\oplus Y$, where $\oplus$ is <exclusive or>. The complete joint <probability distribution> has only these four positive entries:
\Table[
|| $X$
|| $Y$
|| $Z=X\oplus Y$
|| $p(x,y,z)$
| $0$
| $0$
| $0$
| $1/4$
| $0$
| $1$
| $1$
| $1/4$
| $1$
| $0$
| $1$
| $1/4$
| $1$
| $1$
| $0$
| $1/4$
]
{title=Joint probabilities for two independent fair bits and their exclusive or}
Every pair $(x,y)$ has <probability> $1/4=p_X(x)p_Y(y)$, so $I(X;Y)=4(1/4)\log_2 1=0$. But $p_Z(0)=p_Z(1)=1/2$. Given $Z=0$, the pairs $(0,0),(1,1)$ each have conditional <probability> $1/2$, while both conditional marginals are fair. Thus
$$
I(X;Y\mid Z=0)
=\frac12\log_2\frac{1/2}{(1/2)(1/2)}
+\frac12\log_2\frac{1/2}{(1/2)(1/2)}=1.
$$
Given $Z=1$, the pairs $(0,1),(1,0)$ give the same calculation. Therefore
$$
I(X;Y\mid Z)=\frac12\cdot1+\frac12\cdot1=1.
$$
The <conditional entropy> calculation tells the same story: $H(X\mid Z)=1$ because either value of $Z$ leaves $X$ fair, whereas $H(X\mid Y,Z)=0$ because $X=Y\oplus Z$. Learning the <exclusive or> does not reveal either <bit> alone; it reveals whether they agree, so learning one then determines the other.
\Image[/conditional-information-xor.png]
{title=Conditioning on the exclusive or reveals one bit of dependence between independent fair bits}
This example distinguishes <conditional independence> from ordinary <independence>. It also disproves the claim that conditioning always reduces <mutual information>, despite <conditioning reduces entropy>. There is no general ordering between $I(X;Y\mid Z)$ and $I(X;Y)$: the example above increases it from zero to one, while taking $Y=X$ and $Z=X$ for a fair <bit> decreases it from one to zero. The <chain rule for mutual information> explains why both are possible.<6.441 Information Theory (Spring 2016 lecture notes) by Yury Polyanskiy and others/Chapter 2: Information Measures: Mutual Information>[Theorem 2.5]{ref}
<Conditional mutual information> is useful for asking whether an additional observation tells us anything after information already available has been accounted for. Its zero case expresses <conditional independence> exactly; in a <Markov chain>, this is the step that turns the <chain rule for mutual information> into the <data processing inequality>.
Back to article page
