Scope and scale of total correlation

For finite alphabets, $H(X_1,\dots ,X_n)\ge {\max }_{i}H(X_i)$, soFor $n$ identical copies of one fair bit, the upper bound is attained and total correlation is $n-1$ bits. Adding another copy increases the result without introducing a new underlying random choice. The choice of coordinates and their grouping therefore matters when comparing systems of different sizes.

For more general random variables, the relative entropy against the product of the marginals supplies a definition of total correlation that may be infinite. Subtracting differential entropies is justified only when those quantities are finite and the entropy identity is valid; an undefined difference of infinities is not a definition. The finite-alphabet examples above avoid that problem entirely.