= Entropy lower bound from the largest atom
For a finite <probability distribution> with largest atom $\Pi$, its base-two <Shannon entropy> satisfies $H\geq2(1-\Pi)$. If $\Pi\geq1/2$, coarsen the distribution to the indicator of a largest atom: <Shannon entropy> cannot increase under a deterministic function, so $H\geq h_2(\Pi)$. The <concavity of information entropy>[concavity] of <binary entropy> places it above the chord from $(1/2,1)$ to $(1,0)$, giving $h_2(\Pi)\geq2(1-\Pi)$. If $\Pi\leq1/2$, every nonzero probability $p$ has $-\log_2p\geq-\log_2\Pi$, so $H\geq-\log_2\Pi$. The function $-\log_2\Pi-2(1-\Pi)$ vanishes at $1/2$ and has negative derivative on $(0,1/2]$, proving the same bound. Equality in the combined bound occurs for a point mass or a distribution uniform on two points.
Back to article page
