= Cross-entropy
{title2=$H(P,Q)=-\sum_i p_i\log q_i$}
{wiki}
<Cross-entropy> measures the average <logarithmic loss> incurred when outcomes follow one <probability distribution> $P$, but a model assigns <probabilities> using another <probability distribution> $Q$. For a finite <alphabet>, write their <probability mass functions> as $p_i$ and $q_i$. With base-two <logarithms>,
$$
H(P,Q)=-\sum_{i:p_i>0}p_i\log_2 q_i=\mathbb E_P[-\log_2 q(X)].
$$
The expectation is under the true or stipulated source law $P$, rather than the model $Q$. Terms with $p_i=0$ contribute nothing. If $p_i>0$ but $q_i=0$ for any outcome, the <cross-entropy> is $+\infty$: the model rules out an event that can occur. The result is measured in <bits>; <natural logarithms> give <nats>.
The useful identity is <cross-entropy equals entropy plus relative entropy>:
$$
\boxed{H(P,Q)=H(P)+D_{\mathrm{KL}}(P\Vert Q).}
$$
The source's <information entropy> $H(P)$ is the baseline uncertainty. The <Kullback-Leibler divergence> is the extra average cost of a mismatched model, with the true law first and model second. Thus optimizing $Q$ for a fixed $P$ reduces the mismatch, rather than changing the source's own uncertainty. For finite <alphabets>, the minimum is $H(P)$, attained exactly at $Q=P$.
Take a three-symbol source with $P=(1/2,1/4,1/4)$. A matched binary <prefix code> assigns $A\mapsto0$, $B\mapsto10$, $C\mapsto11$, with lengths $(1,2,2)$. A mismatched <prefix code> instead assigns $A\mapsto10$, $B\mapsto11$, $C\mapsto0$, with lengths $(2,2,1)$, corresponding to $Q=(1/4,1/4,1/2)$. Both <prefix codes> decode exactly, but the most common symbol is shorter in the matched code. Under the same source $P$,
$$
\begin{aligned}
H(P)&=\tfrac12\cdot1+\tfrac14\cdot2+\tfrac14\cdot2=\tfrac32,\\
H(P,Q)&=\tfrac12\cdot2+\tfrac14\cdot2+\tfrac14\cdot1=\tfrac74,\\
D_{\mathrm{KL}}(P\Vert Q)&=\tfrac12\log_2 2+\tfrac14\log_2 1+\tfrac14\log_2(1/2)=\tfrac14.
\end{aligned}
$$
\b[The mismatched code costs an extra (1/4) bit per source symbol on average.] No information is lost; the penalty is inefficient representation of this source. The <mismatch penalty in lossless coding> explains when this coding interpretation is exact.
\Image[/cross-entropy-mismatched-prefix-codes.png]
{title=Two lossless prefix codes have different average lengths for the same source}
{description=Both sets of <codewords> are <prefix codes>. The source <probabilities> are fixed at $(1/2,1/4,1/4)$. Each colored contribution is source <probability> times <codeword length>; changing which symbol receives the shortest word increases the total from $3/2$ to $7/4$ <bits> per source symbol.}
<Cross-entropy> is also useful when evaluating a <predictive distribution>. A report that places tiny <probability> on the realized outcome receives a large <logarithmic loss>, even when it usually predicts the most common outcome correctly. The loss convention is the negative of the reward convention in the <logarithmic scoring rule>. Its expected loss is uniquely minimized by the correct <probability distribution>, making it a <strictly proper scoring rule>. <Strictly Proper Scoring Rules, Prediction, and Estimation by Tilmann Gneiting and others> treats this connection in §3.1, Example 3; its discussion uses rewards, so the signs are reversed for the losses here.<Strictly Proper Scoring Rules, Prediction, and Estimation by Tilmann Gneiting and others>{ref}
<I. J. Good>'s <Rational Decisions by I. J. Good> appeared in 1952. Its published summary describes a method for encouraging accurate <probability> estimates and a connection with <information theory>. <Strictly Proper Scoring Rules, Prediction, and Estimation by Tilmann Gneiting and others> traces the <logarithmic scoring rule> at least as far back as this 1952 paper.<Rational Decisions by I. J. Good>{ref}
Back to article page
