Parity constraint and total correlation ($Z=X\oplus Y,\mathrm{TC}(X,Y,Z)=1\text{ bit}$)

Let $X,Y$ be independent fair bits and set $Z=X\oplus Y$, their exclusive or. The four possible triples are $000,011,101,110$, each with probability $1/4$. Every coordinate is fair, and every pair has all four outcomes with equal probability. Thus each pair's mutual information is zero, as in pairwise independence does not imply joint independence.

The full random vector has joint entropy ${\log }_{2}4=2$ bits, while the marginal information entropies sum to three bits. ConsequentlyThe product of the marginals predicts probability $1/8$ for every triple, including the four impossible ones. On each possible triple, the ratio of true to product probability is $(1/4)/(1/8)=2$; averaging its base-two logarithm gives the same one bit of relative entropy.

The example gives a concrete use for joint information measures. A lossless source coding scheme that treats the three coordinates separately needs three bits here. If the decoder knows the integer parity rule, the encoder can send just $X,Y$ and let the decoder reconstruct $Z$ exactly. Pairwise independence does not reveal that saving. This is a stipulated source law and an explicit two-bit code, rather than a promise that arbitrary data can be shortened.

Keep $X,Y$ as independent fair bits, introduce an independent Bernoulli random variable $N$ with $\mathbb{P}(N=1)=\epsilon$, and set $Z=X\oplus Y\oplus N$. The extra bit flips the integer parity rule with probability $\epsilon$. For each even-integer parity triple the probability is $(1-\epsilon )/4$; for each odd-integer parity triple it is $\epsilon /4$.

Every coordinate remains fair, and every pair remains independent for every $0\le \epsilon \le 1$. For example, after fixing $X$, the fair independent $Y$ makes $Y\oplus N$ fair regardless of $\epsilon$. A pair's joint probability distribution therefore does not change as noise destroys the three-way relation.

Knowing $X,Y$, uncertainty in $Z$ is exactly the uncertainty in $N$. The chain rule for information entropy giveswhere $h_2$ is binary entropy. HenceAt $\epsilon =1/4$, the result is approximately $0.188722$ bits, although all three pairwise mutual informations are zero. At $\epsilon =1/2$, all eight triples have probability $1/8$, giving joint independence and zero total correlation. At $\epsilon =1$, the complementary integer parity rule is deterministic again, so total correlation returns to one bit.

The example distinguishes "there is a dependence" from "a pair reveals it." Once $X,Y$ are known together, $Z$ is predictable at zero noise; knowing either one alone leaves a fair bit. Total correlation detects the joint departure from independence, but does not by itself specify which variables cause which others.