= Empirical cross-entropy
{title2=$\widehat H_n(Q)=-n^{-1}\sum_{t=1}^n\log_2 q(X_t)$}
For observations $X_1,\ldots,X_n$ of a <categorical variable>, <empirical cross-entropy> is the average <logarithmic loss> under the reported <probability mass function> $q$. Writing the <empirical distribution> as $\hat p_i=n^{-1}\#\{t:X_t=i\}$ gives
$$
\widehat H_n(Q)=-\frac1n\sum_{t=1}^n\log_2q(X_t)
=-\sum_{i:\hat p_i>0}\hat p_i\log_2q_i=H(\widehat P_n,Q).
$$
Under <independent and identically distributed> observations from $P$ and a fixed model with positive <probability> for each possible outcome, the <law of large numbers> gives convergence to $H(P,Q)$. A finite <empirical distribution> is an estimate of $P$, rather than the true source law; unobserved outcomes need not be impossible. If the same data are used to choose $Q$, evaluating the resulting <empirical cross-entropy> on that data need not estimate its prediction performance on fresh observations without <bias>.
Back to article page
