 Parity constraint and total correlation

Let $X,Y$ be independent fair bits and set $Z=X\oplus Y$, their exclusive or. The four possible triples are $000,011,101,110$, each with probability $1/4$. Every coordinate is fair, and every pair has all four outcomes with equal probability. Thus each pair's mutual information is zero, as in pairwise independence does not imply joint independence.

The full random vector has joint entropy ${\log }_{2}4=2$ bits, while the marginal information entropies sum to three bits. ConsequentlyThe product of the marginals predicts probability $1/8$ for every triple, including the four impossible ones. On each possible triple, the ratio of true to product probability is $(1/4)/(1/8)=2$; averaging its base-two logarithm gives the same one bit of relative entropy.

The example gives a concrete use for joint information measures. A lossless source coding scheme that treats the three coordinates separately needs three bits here. If the decoder knows the integer parity rule, the encoder can send just $X,Y$ and let the decoder reconstruct $Z$ exactly. Pairwise independence does not reveal that saving. This is a stipulated source law and an explicit two-bit code, rather than a promise that arbitrary data can be shortened.