Empirical cross-entropy (${\hat{H}}_n(Q)=-n^{-1}{\sum }_{t=1}^n{\log }_{2}q(X_t)$)

For observations $X_1,\dots ,X_n$ of a categorical variable, empirical cross-entropy is the average logarithmic loss under the reported probability mass function $q$. Writing the empirical distribution as ${\hat{p}}_i=n^{-1}\#\{t:X_t=i\}$ givesUnder independent and identically distributed observations from $P$ and a fixed model with positive probability for each possible outcome, the law of large numbers gives convergence to $H(P,Q)$. A finite empirical distribution is an estimate of $P$, rather than the true source law; unobserved outcomes need not be impossible. If the same data are used to choose $Q$, evaluating the resulting empirical cross-entropy on that data need not estimate its prediction performance on fresh observations without bias.

For a parametric statistical model with probability mass function $q_{\theta }$ and an independent and identically distributed model, minimizing empirical cross-entropy is maximum likelihood estimation. The logarithm is strictly increasing, so maximizing ${\prod }_t{q}_{\theta }(X_t)$ is equivalent to maximizing ${\sum }_t{\log }_2{q}_{\theta }(X_t)$. Negating and dividing by $n$ gives the displayed minimization. Restricting the model family can prevent the optimum from matching the empirical distribution, and a finite optimizing parameter need not exist. This algebra does not by itself establish accuracy on new data.^[ref]