Information entropy ($H(X)$)

For a discrete random variable with probabilities $p_i$, its Shannon information entropy is

If one probability in a $k$-point distribution equals $t$, normalize the remaining $k-1$ probabilities to $q$. Then $H=h(t)+(1-t)H(q)\le h(t)+(1-t){\log }_2(k-1)$. Equality for $t<1$ means that the remaining probabilities are equal. This combines the binary entropy of the distinguished event with the largest residual Shannon entropy.

For a finite-valued random variable, an epsilon-sufficient set $S$ retains probability at least $1-ϵ$. This notion is about covering probability mass and is unrelated to a sufficient statistic. For an IID random variable source, any such set of words with fixed $ϵ<1$ requires exponentially many words at a rate approaching the Shannon entropy.

For finite-alphabet independent and identically distributed random variables, fixed $0\le ϵ<1$ and $\delta >0$, every epsilon-sufficient set $S_n$ of length-$n$ words satisfies $|S_n|\ge (1-ϵ)2^{n(H-\delta )}/2$ for sufficiently large $n$. Intersect $S_n$ with a typical set whose excluded mass is at most $(1-ϵ)/2$. The intersection retains that much probability and each word has probability at most $2^{-n(H-\delta )}$.

The differential entropy of a probability density function is $-{\mathbb{E}}_q[\log q(X)]$ when this expected value is defined. Unlike discrete information entropy, it can be negative and depends on the coordinates and reference measure. It is the entropy term in the evidence lower bound for continuous variational inference.

For a discrete random variable with largest point probability $p_{\max }$, its min-entropy is

For a discrete random variable, $H(X)\ge H_{\infty }(X)$. Indeed, every point probability satisfies $p_x\le p_{\max }$, so averaging $-\log p_x\ge -\log p_{\max }$ proves the inequality.

The varentropy of a discrete random variable with mass function $P$ is the variance of its information content:

The binary entropy function is

For discrete random variables,Iterating gives $H(X_1,\dots ,X_n)={\sum }_{i}H(X_i\mid X_1,\dots ,X_{i-1})$.

The chain rule for information entropy and conditioning reduces entropy givewith equality exactly when $X$ and $Y$ are independent.

For probability mass functions $P_1,\dots ,P_m$ and convex weights ${\alpha }_i$, information entropy satisfiesThis follows from the concavity of $-t\log t$.

Among probability distributions on a finite set of $m$ elements, the uniform distribution uniquely maximizes information entropy, attaining $\log m$.

An information entropy on at most $m$ outcomes is at most ${\log }_{2}m$, with equality for the uniform distribution on all $m$ outcomes. For the uniform reference $u$, the Kullback-Leibler divergence is $D(p\Vert u)={\log }_{2}m-H(p)\ge 0$, proving the bound.

Among distributions on $\{0,1,2,\dots \}$ with fixed expected value $\mu$, the geometric distribution with success probability $1/(1+\mu )$ uniquely maximizes information entropy. Its entropy is