Relative entropy to a product law ($D(P\Vert {⨂}_i{Q}_i)=\mathrm{TC}(P)+{\sum }_{i}D(P_i\Vert Q_i)$)

Let $P$ be a joint probability distribution on a finite product alphabet, with marginals $P_i$. Compare it with a product model $Q={⨂}_i{Q}_i$, assuming $Q_i(x_i)>0$ wherever $P_i(x_i)>0$. Adding and subtracting the logarithm of ${\prod }_i{P}_i(x_i)$ givesThe first term is the unavoidable error from discarding dependence; the other terms are errors in the individual marginal probability distributions. Averaging a term that depends only on $x_i$ under $P$ is the same as averaging it under $P_i$, which proves the identity directly.

Among product laws, the choice $Q_i=P_i$ minimizes this relative entropy, leaving exactly total correlation. The related cross-entropy equals entropy plus relative entropy identity interprets this as extra average logarithmic loss from using the independent model. It is a statement about a specified joint law and loss function, not a claim that fitting a finite sample automatically recovers that law.