= Mutual information
{title2=$I(X;Y)$}
{wiki}
For finite <discrete random variables>, mutual information is the reduction of <information entropy> provided by observing the other variable:
$$
I(X;Y)=H(Y)-H(Y\mid X)
=H(X)+H(Y)-H(X,Y).
$$
The last expression makes the symmetry $I(X;Y)=I(Y;X)$ visible. It is nonnegative and is zero exactly when $X$ and $Y$ are <independent random variables>. One way to see both facts is
$$
I(X;Y)=\sum_{x,y:p(x,y)>0}p(x,y)\log_2\frac{p(x,y)}{p_X(x)p_Y(y)}
=D(p_{X,Y}\Vert p_Xp_Y),
$$
where $D$ is the <Kullback-Leibler divergence> with base-two <logarithms>. The <Gibbs inequality> gives nonnegativity; equality means that the joint <probability distribution> factors. The statement extends to more general distributions via <relative entropy>, but differences of infinite <information entropies> must not be used as if they were finite numbers.
A <binary symmetric channel> gives a useful example. Let $X$ be a fair <bit>, let $N$ be an <independent> <Bernoulli random variable> with error <probability> $q$, and set $Y=X\oplus N$, where $\oplus$ is <exclusive or>. The output is also a fair <bit>, but knowing the input leaves just the error uncertainty:
$$
H(Y)=1,\qquad H(Y\mid X)=h_2(q),\qquad I(X;Y)=1-h_2(q).
$$
Here $h_2$ is <binary entropy>. At $q=0.1$, $h_2(q)\approx0.469$, so the output carries about $0.531$ <bits> of <mutual information> about the input per use. This is an average uncertainty reduction, not a claim that each observed symbol can be divided into fractional physical <bits>.
\Image[/binary-channel-information.png]
{title=Binary entropy and retained information distinguish noisy transmission from predictable bit inversion}
At $q=1/2$, every output is equally likely for either input, so $I(X;Y)=0$. At $q=1$, the output is always the inverted input: knowing this rule lets the receiver recover the input exactly, so $I(X;Y)=1$ again. A high raw error rate need not mean low <mutual information> when the errors are predictable. For this particular channel the fair input is optimal, giving the <binary symmetric channel capacity> $1-h_2(q)$.
<Mutual information> detects dependence that ordinary <correlation> can miss. Let $X$ be uniform on $\{-1,0,1\}$ and $Y=X^2$. Symmetry gives $\mathbb E[X]=\mathbb E[XY]=0$, hence zero <covariance> and zero <correlation>, while $Y$ is completely determined by $X$. Thus $I(X;Y)=H(Y)=h_2(1/3)>0$. This distinction is useful whenever a relationship is nonlinear: knowing one variable can reveal another without a linear trend.
Back to article page
