= Cross-entropy and maximum likelihood
{title2=$\arg\min_\theta\widehat H_n(Q_\theta)=\arg\max_\theta\prod_tq_\theta(X_t)$}
For a <parametric statistical model> with <probability mass function> $q_\theta$ and an <independent and identically distributed> model, minimizing <empirical cross-entropy> is <maximum likelihood estimation>. The <logarithm> is strictly increasing, so maximizing $\prod_tq_\theta(X_t)$ is equivalent to maximizing $\sum_t\log_2q_\theta(X_t)$. Negating and dividing by $n$ gives the displayed minimization. Restricting the model family can prevent the optimum from matching the <empirical distribution>, and a finite optimizing parameter need not exist. This algebra does not by itself establish accuracy on new data.<Strictly Proper Scoring Rules, Prediction, and Estimation by Tilmann Gneiting and others>[§1, printed p. 360: optimum-score estimation and maximum likelihood]{ref}
Back to article page
