= Expected relative entropy decomposition
{title2=$\mathbb E D(Q\|r)=\mathbb E D(Q\|p)+D(p\|r)$}
Let $Q$ be a random <probability mass function> on a <finite alphabet>, let $p=\mathbb E Q$, and suppose a fixed reference $r$ is positive on the <probability support> of $p$. Then $Q$ vanishes <almost surely> wherever $p=0$, and
$$
D(Q\|r)-D(Q\|p)=\sum_{i:p_i>0}Q_i\log_2\frac{p_i}{r_i}.
$$
Taking <expected values> yields the displayed decomposition. In particular it applies to an unbiased <empirical distribution>. It separates sampling fluctuation from reference mismatch: even perfect recovery of $p$ leaves the deterministic <relative entropy> $D(p\|r)$. A bound comparing different references must account for this term.
Back to article page
