Maximum entropy distribution on a finite set

Among probability distributions on a finite set of $m$ elements, the uniform distribution uniquely maximizes information entropy, attaining $\log m$.

For a finite list of real values $y_i$, maximize information entropy $H=-{\sum }_i{p}_i\log p_i$ with $p_i>0$, ${\sum }_i{p}_i=1$, and ${\sum }_i{p}_i{y}_i=Y$. Lagrange multipliers giveThe constrained second variation is $-{\sum }_i(\delta p_i{)}^2/p_i<0$ for any nonzero tangent perturbation, proving uniqueness of the entropy maximum whenever a feasible interior solution exists. The mean has derivative $-{\mathrm{Var}}_b(y)$, so for nonconstant $y_i$ it decreases strictly with $b$ and ranges between the largest and smallest $y_i$ as $b$ ranges over the real line. Thus a finite $b$ exists uniquely exactly when ${\min }_i{y}_i<Y<{\max }_i{y}_i$.

At an extreme mean, strictly positive probabilities are impossible unless all $y_i$ coincide. If all $y_i=y_0$, the mean constraint is redundant when $Y=y_0$, the maximum is uniform, and $b$ is not identifiable. This finite exponential family is an exponential tilting of the uniform distribution.

An information entropy on at most $m$ outcomes is at most ${\log }_{2}m$, with equality for the uniform distribution on all $m$ outcomes. For the uniform reference $u$, the Kullback-Leibler divergence is $D(p\Vert u)={\log }_{2}m-H(p)\ge 0$, proving the bound.