= Finite-alphabet entropy continuity bound
{title2=$|H(p)-H(q)|\leq d\log_2(N/d),\quad d=\|p-q\|_1\leq1/2$}
For <probability mass functions> on $N$ symbols, set $d_i=|p_i-q_i|$ and $d=\sum_id_i\leq1/2$. The <entropy summand> bound gives $|f(p_i)-f(q_i)|\leq f(d_i)$. The <triangle inequality> and <Jensen's inequality> give
$$
|H(p)-H(q)|\leq\sum_if(d_i)\leq Nf(d/N)=d\log_2(N/d).
$$
At $d=0$, use the continuous value zero. This proves uniform continuity of <Shannon entropy> on a fixed <finite alphabet>; the <alphabet> size appears explicitly in the bound.
Back to article page
