 Empirical cross-entropy

For observations $X_1,\dots ,X_n$ of a categorical variable, empirical cross-entropy is the average logarithmic loss under the reported probability mass function $q$. Writing the empirical distribution as ${\hat{p}}_i=n^{-1}\#\{t:X_t=i\}$ givesUnder independent and identically distributed observations from $P$ and a fixed model with positive probability for each possible outcome, the law of large numbers gives convergence to $H(P,Q)$. A finite empirical distribution is an estimate of $P$, rather than the true source law; unobserved outcomes need not be impossible. If the same data are used to choose $Q$, evaluating the resulting empirical cross-entropy on that data need not estimate its prediction performance on fresh observations without bias.