Cross-entropy ($H(P,Q)=-{\sum }_i{p}_i\log q_i$)

Cross-entropy measures the average logarithmic loss incurred when outcomes follow one probability distribution $P$, but a model assigns probabilities using another probability distribution $Q$. For a finite alphabet, write their probability mass functions as $p_i$ and $q_i$. With base-two logarithms,The expectation is under the true or stipulated source law $P$, rather than the model $Q$. Terms with $p_i=0$ contribute nothing. If $p_i>0$ but $q_i=0$ for any outcome, the cross-entropy is $+\infty$: the model rules out an event that can occur. The result is measured in bits; natural logarithms give nats.

The useful identity is cross-entropy equals entropy plus relative entropy:The source's information entropy $H(P)$ is the baseline uncertainty. The Kullback-Leibler divergence is the extra average cost of a mismatched model, with the true law first and model second. Thus optimizing $Q$ for a fixed $P$ reduces the mismatch, rather than changing the source's own uncertainty. For finite alphabets, the minimum is $H(P)$, attained exactly at $Q=P$.

Take a three-symbol source with $P=(1/2,1/4,1/4)$. A matched binary prefix code assigns $A\mapsto 0$, $B\mapsto 10$, $C\mapsto 11$, with lengths $(1,2,2)$. A mismatched prefix code instead assigns $A\mapsto 10$, $B\mapsto 11$, $C\mapsto 0$, with lengths $(2,2,1)$, corresponding to $Q=(1/4,1/4,1/2)$. Both prefix codes decode exactly, but the most common symbol is shorter in the matched code. Under the same source $P$,The mismatched code costs an extra (1/4) bit per source symbol on average. No information is lost; the penalty is inefficient representation of this source. The mismatch penalty in lossless coding explains when this coding interpretation is exact.

Cross-entropy is also useful when evaluating a predictive distribution. A report that places tiny probability on the realized outcome receives a large logarithmic loss, even when it usually predicts the most common outcome correctly. The loss convention is the negative of the reward convention in the logarithmic scoring rule. Its expected loss is uniquely minimized by the correct probability distribution, making it a strictly proper scoring rule. Strictly Proper Scoring Rules, Prediction, and Estimation by Tilmann Gneiting and others treats this connection in §3.1, Example 3; its discussion uses rewards, so the signs are reversed for the losses here.^[ref]

I. J. Good's Rational Decisions by I. J. Good appeared in 1952. Its published summary describes a method for encouraging accurate probability estimates and a connection with information theory. Strictly Proper Scoring Rules, Prediction, and Estimation by Tilmann Gneiting and others traces the logarithmic scoring rule at least as far back as this 1952 paper.^[ref]

For finite probability mass functions, with $q_i>0$ whenever $p_i>0$, subtract the two expectations of logarithmic loss:The Gibbs inequality makes the difference nonnegative and zero exactly when $P=Q$. If $Q$ assigns zero probability to an outcome with positive probability under $P$, both the cross-entropy and the Kullback-Leibler divergence are infinite. The finite information entropy still leaves the displayed identity valid in the extended sense. In more general settings, undefined differences such as $\infty -\infty$ cannot be used as a derivation.

Suppose a binary prefix code has integer codeword lengths ${\ell }_i=-{\log }_2{q}_i$. Under a source with probability mass function $p_i$, its average codeword length is exactly cross-entropy, ${\sum }_i{p}_i{\ell }_i=H(P,Q)$. The Kraft inequality is satisfied because ${\sum }_i{2}^{-{\ell }_i}={\sum }_i{q}_i=1$. This exact interpretation requires suitable dyadic rational probabilities; a general $-{\log }_2{q}_i$ is not an integer codeword length. For positive $q_i$, a prefix code with lengths $\lceil -{\log }_2{q}_i\rceil$ exists by the Kraft inequality and has average length between $H(P,Q)$ and $H(P,Q)+1$. The ceiling function causes the one-symbol rounding overhead. The cross-entropy term describes mismatch overhead; those are distinct costs.

Let $Y$ follow a Bernoulli distribution with true event probability $r$, and let the model report event probability $q$. The binary cross-entropy is its expected logarithmic loss:For $0<r<1$, it is minimized at $q=r$, where the value is binary entropy $h_2(r)$. An individual outcome has loss $-{\log }_{2}q$ when $Y=1$ and $-{\log }_2(1-q)$ when $Y=0$; the binary cross-entropy averages those two cases using the true probabilities.

For $r=1/4$, truthful reporting gives $h_2(1/4)\approx 0.811278$ bits. Reporting $q=1/2$ gives one bit. Reporting $q=0.01$ givesBoth $q=1/4$ and $q=0.01$ select zero if one predicts only the more likely binary outcome. Their predicted probabilities are very different, however, and cross-entropy reveals the cost of underestimating the event that occurs a quarter of the time. Endpoint losses are computed using the realized outcome rather than multiplying zero by an undefined logarithm.

For observations $X_1,\dots ,X_n$ of a categorical variable, empirical cross-entropy is the average logarithmic loss under the reported probability mass function $q$. Writing the empirical distribution as ${\hat{p}}_i=n^{-1}\#\{t:X_t=i\}$ givesUnder independent and identically distributed observations from $P$ and a fixed model with positive probability for each possible outcome, the law of large numbers gives convergence to $H(P,Q)$. A finite empirical distribution is an estimate of $P$, rather than the true source law; unobserved outcomes need not be impossible. If the same data are used to choose $Q$, evaluating the resulting empirical cross-entropy on that data need not estimate its prediction performance on fresh observations without bias.

For a parametric statistical model with probability mass function $q_{\theta }$ and an independent and identically distributed model, minimizing empirical cross-entropy is maximum likelihood estimation. The logarithm is strictly increasing, so maximizing ${\prod }_t{q}_{\theta }(X_t)$ is equivalent to maximizing ${\sum }_t{\log }_2{q}_{\theta }(X_t)$. Negating and dividing by $n$ gives the displayed minimization. Restricting the model family can prevent the optimum from matching the empirical distribution, and a finite optimizing parameter need not exist. This algebra does not by itself establish accuracy on new data.^[ref]