= Solution
For the <maximum entropy distribution on the nonnegative integers>, use the <geometric distribution> counting failures before the first success. Write $r=a/(a+1)$ and define
$$
\boxed{g_i=(1-r)r^i=\frac1{a+1}\left(\frac a{a+1}\right)^i,\qquad i=0,1,\ldots.}
$$
The <geometric series> gives $\sum_i g_i=1$. <Termwise differentiation of a power series> applied to that <geometric series> for $0<r<1$ gives $\sum_iir^i=r/(1-r)^2$, hence $\sum_iig_i=r/(1-r)=a$.
For any competing <probability mass function> $p$ of mean $a$, its <cross-entropy> against $g$ is
$$
-\sum_{i\geq0}p_i\log_2g_i
=\log_2(a+1)+a\log_2\frac{a+1}{a}
=(a+1)\log_2(a+1)-a\log_2a.
$$
This is finite. The fact that <finite cross entropy bounds countable information entropy> must be justified without assuming in advance that $H(p)$ is finite. Use $\ln u\leq u-1$ with $u=g_i/p_i$ when $p_i>0$. Including zero terms by their limiting convention yields, for every finite $N$,
$$
\sum_{i=0}^N-p_i\log_2p_i\leq\sum_{i=0}^N-p_i\log_2g_i+\frac1{\ln2}\sum_{i=0}^N(g_i-p_i).
$$
Let $N\to\infty$. The <information entropy> partial sums increase, the <cross-entropy> partial sums converge to the finite value above, and the final mass difference tends to zero. This proves finiteness and the upper bound. We may now write
$$
D(p\|g)=H(g)-H(p)\geq0.
$$
<Relative entropy nonnegativity> has equality exactly when $p=g$. Therefore the displayed geometric <probability mass function> is the \b[unique maximizer], and
$$
\boxed{H_{\max}=(a+1)\log_2(a+1)-a\log_2a\ \text{bits}.}
$$
The mean constraint is essential: without it, <uniform distributions> on longer initial intervals have unbounded <information entropy>.
Back to article page
