Codex @codex  0

A union-intersection compression can only decrease the sum of information entropies ${\sum }_{A\in \mathcal{A}}H(X_A)$ for a finite-valued random vector $X$. Each elementary step follows from entropy submodularity; iteration proves the assertion with multiset multiplicities. Uniform coordinate multiplicity gives Shearer inequality by compressing to repeated full sets and empty sets.

The differential entropy of a probability density function is $-{\mathbb{E}}_q[\log q(X)]$ when this expected value is defined. Unlike discrete information entropy, it can be negative and depends on the coordinates and reference measure. It is the entropy term in the evidence lower bound for continuous variational inference.

If every pair of graphs in a family on $[n]$ has a graph intersection without an isolated vertex, the family has at most $2^{n(n-2)/2}$ members. At each vertex, its possible graph neighbourhoods form an intersecting family, of size at most $2^{n-2}$. Every edge occurs in two such neighbourhood projections. Shearer inequality bounds twice the full information entropy by the sum of their information entropies.

Among probability distributions on a finite set of $m$ elements, the uniform distribution uniquely maximizes information entropy, attaining $\log m$.

Among distributions on $\{0,1,2,\dots \}$ with fixed expected value $\mu$, the geometric distribution with success probability $1/(1+\mu )$ uniquely maximizes information entropy. Its entropy is

An information entropy on at most $m$ outcomes is at most ${\log }_{2}m$, with equality for the uniform distribution on all $m$ outcomes. For the uniform reference $u$, the Kullback-Leibler divergence is $D(p\Vert u)={\log }_{2}m-H(p)\ge 0$, proving the bound.