Entropy lower bound from the largest atom

ID: entropy-lower-bound-from-the-largest-atom

For a finite probability distribution with largest atom , its base-two Shannon entropy satisfies . If , coarsen the distribution to the indicator of a largest atom: Shannon entropy cannot increase under a deterministic function, so . The concavity of binary entropy places it above the chord from to , giving . If , every nonzero probability has , so . The function vanishes at and has negative derivative on , proving the same bound. Equality in the combined bound occurs for a point mass or a distribution uniform on two points.

New to topics? Read the docs here!