= Relative entropy to a product law
{title2=$D(P\Vert\bigotimes_i Q_i)=\operatorname{TC}(P)+\sum_iD(P_i\Vert Q_i)$}
Let $P$ be a joint <probability distribution> on a finite product <alphabet>, with marginals $P_i$. Compare it with a product model $Q=\bigotimes_iQ_i$, assuming $Q_i(x_i)>0$ wherever $P_i(x_i)>0$. Adding and subtracting the <logarithm> of $\prod_iP_i(x_i)$ gives
$$
\boxed{D(P\Vert Q)=\operatorname{TC}(P)+\sum_i D(P_i\Vert Q_i).}
$$
The first term is the unavoidable error from discarding dependence; the other terms are errors in the individual marginal <probability distributions>. Averaging a term that depends only on $x_i$ under $P$ is the same as averaging it under $P_i$, which proves the identity directly.
Among product laws, the choice $Q_i=P_i$ minimizes this <relative entropy>, leaving exactly <total correlation>. The related <cross-entropy equals entropy plus relative entropy> identity interprets this as extra average <logarithmic loss> from using the independent model. It is a statement about a specified joint law and loss function, not a claim that fitting a finite sample automatically recovers that law.
Back to article page
