= Cross-entropy equals entropy plus relative entropy
{title2=$H(P,Q)-H(P)=D_{\mathrm{KL}}(P\Vert Q)$}
For finite <probability mass functions>, with $q_i>0$ whenever $p_i>0$, subtract the two <expectations> of <logarithmic loss>:
$$
\begin{aligned}
H(P,Q)-H(P)
&=\sum_{i:p_i>0}p_i(\log_2 p_i-\log_2 q_i)\\
&=\sum_{i:p_i>0}p_i\log_2\frac{p_i}{q_i}
=D_{\mathrm{KL}}(P\Vert Q).
\end{aligned}
$$
The <Gibbs inequality> makes the difference nonnegative and zero exactly when $P=Q$. If $Q$ assigns zero <probability> to an outcome with positive <probability> under $P$, both the <cross-entropy> and the <Kullback-Leibler divergence> are infinite. The finite <information entropy> still leaves the displayed identity valid in the extended sense. In more general settings, undefined differences such as $\infty-\infty$ cannot be used as a derivation.
Back to article page
