= Subadditivity of information entropy
For discrete <random variables> $X,Y$, <Shannon entropy> satisfies
$$
H(X,Y)\leq H(X)+H(Y).
$$
When the two marginal <Shannon entropies> are finite, the <chain rule for information entropy> and <conditioning reduces entropy> prove the inequality, and the difference is the <mutual information> $I(X;Y)$. Thus equality holds exactly when $X$ and $Y$ are <independent random variables>.
For example, let $X=Y$ be a fair <Bernoulli random variable>. Then $H(X,Y)=1$ <bit>, while $H(X)+H(Y)=2$ <bits>: the pair does not contain two independent copies of the information. The inequality also holds with extended nonnegative values, but equality of two infinite values does not characterize <independence>.
Back to article page
