= Information entropy
{title2=$H(X)$}
= Shannon entropy
{c}
{synonym}
For a finite <discrete random variable> with <probability mass function> $p$, its information entropy is the <expected value> of its <information content>:
$$
H(X)=-\sum_{x:p(x)>0}p(x)\log_2p(x)
=\mathbb E[-\log_2p(X)].
$$
Base-two <logarithms> give <bits>; choosing a different base rescales the units. The convention $0\log_2 0=0$ is justified by a <limit>, so outcomes of zero <probability> contribute nothing. The definition also applies to countable distributions, but the sum may be infinite.
A fair <bit> has $H=1$. A certain outcome has $H=0$. A <Bernoulli random variable> that is one with <probability> $0.1$ has $H=h_2(0.1)\approx0.469$ <bits>, because most of its outcomes are predictable. A rare outcome still has large <information content>: the outcome of <probability> $0.1$ contributes $-\log_2(0.1)\approx3.322$ <bits> of surprise when it occurs. \b[Entropy is the average surprise, rather than the surprise of each outcome.]
For $m$ possible outcomes, $0\leq H(X)\leq\log_2m$, with the upper bound attained exactly by the <uniform distribution on a finite set>. The <Gibbs inequality> proves this by comparing $p$ with the uniform distribution: $D(p\Vert u)=\log_2m-H(X)\geq0$. Thus increasing predictability reduces the required average description length. The four-symbol <prefix code> in <information theory> attains $H=7/4$ <bits> exactly; for general probabilities, integer codeword lengths prevent such a perfect match for every individual symbol. <Shannon's source coding theorem> explains how coding sufficiently long independent blocks approaches the entropy per symbol.
The <logarithm> is especially natural because <independent> outcomes have additive surprise: $-\log_2(pq)=-\log_2p-\log_2q$. Taking <expected values> gives additive <information entropy> for <independent random variables>; for dependent ones, the <chain rule for information entropy> replaces the second term by <conditional entropy>. This is why the same quantity can govern both a single observation and a long message.
Back to article page