= Finite cross entropy bounds countable information entropy
{title2=$H(p)\leq-\sum_i p_i\log_2g_i$}
Let $p$ and $g$ be <probability mass functions> on a countable <alphabet>, with $g_i>0$ whenever $p_i>0$. If the <cross-entropy> $C=-\sum_i p_i\log_2g_i$ is finite, then $H(p)$ is finite, $H(p)\leq C$, and $D(p\|g)=C-H(p)$. To avoid assuming the desired finiteness, first apply $\ln u\leq u-1$ with $u=g_i/p_i$ to each positive $p_i$. For a finite initial segment,
$$
\sum_{i=0}^N-p_i\log_2p_i\leq\sum_{i=0}^N-p_i\log_2g_i+\frac{\sum_{i=0}^N(g_i-p_i)}{\ln2}.
$$
Zero coordinates satisfy this inequality too. <Monotone convergence> of the nonnegative <information entropy> summands and convergence of both total masses give $H(p)\leq C<\infty$. Both logarithmic sums are then finite, justifying their difference and the <relative entropy> identity. This supplies a rigorous route to the <maximum entropy distribution on the nonnegative integers> without presupposing finite <information entropy> for every competing law.
Back to article page
