= Total correlation
{title2=$\operatorname{TC}(X_1,\ldots,X_n)$}
{wiki}
<Total correlation> measures how much dependence is lost when a joint <probability distribution> is replaced by the product of its marginal <probability distributions>. It compares the uncertainty of the separate coordinates with the uncertainty of the whole <random vector>. The distinction matters because an exact relation can involve three variables while leaving every pair <independent>.
For <random variables> $X_1,\ldots,X_n$ with finite <alphabets>, define
$$
\boxed{\operatorname{TC}(X_1,\ldots,X_n)
=\sum_{i=1}^n H(X_i)-H(X_1,\ldots,X_n)
=D\!\left(P_{X_1,\ldots,X_n}\middle\Vert\bigotimes_{i=1}^n P_{X_i}\right).}
$$
Here $H$ is <information entropy>, the entropy of the full <random vector> is <joint entropy>, and $D$ is <relative entropy>. Base-two <logarithms> give <bits>; <natural logarithms> give <nats>. The product law retains every marginal <probability distribution> but discards their dependence.<Information Theoretical Analysis of Multivariate Correlation by Satosi Watanabe>[Abstract, printed p. 66: total correlation for the partition into individual variables]{ref} A positive joint <probability> guarantees positive marginal <probabilities> at that outcome, so the displayed <relative entropy> is finite in this finite-alphabet setting.
For two variables, <total correlation> is exactly <mutual information>. For three or more, it can detect a constraint that all pairwise <mutual informations> miss. <Relative entropy nonnegativity> gives $\operatorname{TC}\geq0$, with equality exactly when the coordinates are jointly <independent random variables>.
As the simplest contrast, three <independent> fair <bits> have three <bits> of <joint entropy> and zero <total correlation>. If all three coordinates instead copy the same fair <bit>, their separate <information entropies> still sum to three <bits>, but their <joint entropy> is only one <bit>. Their <total correlation> is therefore two <bits>. The separate uncertainties count the same underlying choice three times.
Back to article page
