= Scope and scale of total correlation
For finite <alphabets>, $H(X_1,\ldots,X_n)\geq\max_iH(X_i)$, so
$$
0\leq\operatorname{TC}(X_1,\ldots,X_n)
\leq\sum_iH(X_i)-\max_iH(X_i).
$$
For $n$ identical copies of one fair <bit>, the upper bound is attained and <total correlation> is $n-1$ <bits>. Adding another copy increases the result without introducing a new underlying random choice. The choice of coordinates and their grouping therefore matters when comparing systems of different sizes.
For more general <random variables>, the <relative entropy> against the product of the marginals supplies a definition of <total correlation> that may be infinite. Subtracting <differential entropies> is justified only when those quantities are finite and the entropy identity is valid; an undefined difference of infinities is not a definition. The finite-<alphabet> examples above avoid that problem entirely.
Back to article page
