Cross-entropy and maximum likelihood

ID: cross-entropy-and-maximum-likelihood

For a parametric statistical model with probability mass function and an independent and identically distributed model, minimizing empirical cross-entropy is maximum likelihood estimation. The logarithm is strictly increasing, so maximizing is equivalent to maximizing . Negating and dividing by gives the displayed minimization. Restricting the model family can prevent the optimum from matching the empirical distribution, and a finite optimizing parameter need not exist. This algebra does not by itself establish accuracy on new data.[ref]

New to topics? Read the docs here!