Information entropy ($H(X)$)

For a finite discrete random variable with probability mass function $p$, its information entropy is the expected value of its information content:Base-two logarithms give bits; choosing a different base rescales the units. The convention $0{\log }_{2}0=0$ is justified by a limit, so outcomes of zero probability contribute nothing. The definition also applies to countable distributions, but the sum may be infinite.

A fair bit has $H=1$. A certain outcome has $H=0$. A Bernoulli random variable that is one with probability $0.1$ has $H=h_2(0.1)\approx 0.469$ bits, because most of its outcomes are predictable. A rare outcome still has large information content: the outcome of probability $0.1$ contributes $-{\log }_2(0.1)\approx 3.322$ bits of surprise when it occurs. Entropy is the average surprise, rather than the surprise of each outcome.

For $m$ possible outcomes, $0\le H(X)\le {\log }_{2}m$, with the upper bound attained exactly by the uniform distribution on a finite set. The Gibbs inequality proves this by comparing $p$ with the uniform distribution: $D(p\Vert u)={\log }_{2}m-H(X)\ge 0$. Thus increasing predictability reduces the required average description length. The four-symbol prefix code in information theory attains $H=7/4$ bits exactly; for general probabilities, integer codeword lengths prevent such a perfect match for every individual symbol. Shannon's source coding theorem explains how coding sufficiently long independent blocks approaches the entropy per symbol.

The logarithm is especially natural because independent outcomes have additive surprise: $-{\log }_2(pq)=-{\log }_{2}p-{\log }_{2}q$. Taking expected values gives additive information entropy for independent random variables; for dependent ones, the chain rule for information entropy replaces the second term by conditional entropy. This is why the same quantity can govern both a single observation and a long message.

For a random finite binary string $M$ of length $L$, conditioning on $L=\ell$ leaves at most $2^{\ell }$ possible strings. Thus its conditional entropy is at most $\ell$. Since the length is a function of the string, the chain rule for information entropy givesThe term $H(L)$ cannot generally be removed. For concatenations of $m$ words from a fixed finite decipherable code with maximum length $d$, it is at most ${\log }_2(md+1)$. Decipherability preserves the Shannon entropy of the source block. For an independent source block, dividing the resulting inequality by $m$ and taking $m\to \infty$ proves the expected codeword length is at least the one-symbol Shannon entropy.

For a finite probability distribution with largest atom $\Pi$, its base-two Shannon entropy satisfies $H\ge 2(1-\Pi )$. If $\Pi \ge 1/2$, coarsen the distribution to the indicator of a largest atom: Shannon entropy cannot increase under a deterministic function, so $H\ge h_2(\Pi )$. The concavity of binary entropy places it above the chord from $(1/2,1)$ to $(1,0)$, giving $h_2(\Pi )\ge 2(1-\Pi )$. If $\Pi \le 1/2$, every nonzero probability $p$ has $-{\log }_{2}p\ge -{\log }_2\Pi$, so $H\ge -{\log }_2\Pi$. The function $-{\log }_2\Pi -2(1-\Pi )$ vanishes at $1/2$ and has negative derivative on $(0,1/2]$, proving the same bound. Equality in the combined bound occurs for a point mass or a distribution uniform on two points.

Let $p$ and $g$ be probability mass functions on a countable alphabet, with $g_i>0$ whenever $p_i>0$. If the cross-entropy $C=-{\sum }_i{p}_i{\log }_2{g}_i$ is finite, then $H(p)$ is finite, $H(p)\le C$, and $D(p\Vert g)=C-H(p)$. To avoid assuming the desired finiteness, first apply $\ln u\le u-1$ with $u=g_i/p_i$ to each positive $p_i$. For a finite initial segment,Zero coordinates satisfy this inequality too. Monotone convergence of the nonnegative information entropy summands and convergence of both total masses give $H(p)\le C<\infty$. Both logarithmic sums are then finite, justifying their difference and the relative entropy identity. This supplies a rigorous route to the maximum entropy distribution on the nonnegative integers without presupposing finite information entropy for every competing law.

For probability mass functions on $N$ symbols, set $d_i=|p_i-q_i|$ and $d={\sum }_i{d}_i\le 1/2$. The entropy summand bound gives $|f(p_i)-f(q_i)|\le f(d_i)$. The triangle inequality and Jensen's inequality giveAt $d=0$, use the continuous value zero. This proves uniform continuity of Shannon entropy on a fixed finite alphabet; the alphabet size appears explicitly in the bound.

The contribution of a symbol with probability $x$ to Shannon entropy, with $f(0)=0$. It is a concave function because $f^{\prime \prime }(x)=-1/(x\ln 2)<0$ on $(0,1)$ and its endpoint extension is continuous. Its sharp bound for pairs exactly distance $v$ apart in $[0,1]$ isTo prove this, the increment $f(t+v)-f(t)$ decreases with $t$, from $f(v)$ to $-f(1-v)$. For $v\le 1/2$, the maximum is $f(v)$: the difference $f(v)-f(1-v)$ vanishes at zero and one half and has second derivative $(2v-1)/(v(1-v)\ln 2)\le 0$ between them. Hence it is nonnegative by concavity.

Joint entropy is the information entropy of an entire random vector, treating its coordinates together as one outcome. For random variables with finite alphabets and joint probability mass function $p$,It is measured in bits. The distinction from separate information entropies is whether shared uncertainty is counted once or repeatedly. Two copies of the same fair bit have joint entropy one bit, since only $00$ and $11$ can occur. Two independent fair bits have joint entropy two bits, since all four pairs are possible and equiprobable.

For a less symmetric example, assign probabilities $1/2,1/4,1/4$ to $(X,Y)=(0,0),(1,0),(1,1)$ respectively. ThenHere $X$ is fair, while $\mathbb{P}(Y=1)=1/4$. Thus $H(X)=1$ and $H(Y)=h_2(1/4)\approx 0.811278$ bits, where $h_2$ is binary entropy. Adding the separate information entropies would give approximately $1.811278$ bits, overcounting by the mutual information $I(X;Y)\approx 0.311278$ bits.

The chain rule for information entropy makes the shared information explicit:In this example $Y$ is certain given $X=0$ and fair given $X=1$, so its average conditional entropy is $H(Y\mid X)=\frac{1}{2}\cdot 0+\frac{1}{2}\cdot 1=1/2$ bit. Sending the first coordinate changes how much uncertainty remains in the second. A prefix code with codewords 0, 10, 11 for the three possible pairs has average length $3/2$ bits, exactly the joint entropy in this stipulated law.

For finite alphabets,The lower bound follows because knowing the full random vector determines every coordinate; the upper bound follows from conditioning reduces entropy in the chain rule for information entropy. Equality in the upper bound means joint independence. The difference from that upper bound is total correlation, whose noisy parity total correlation example shows why separate pairwise comparisons need not find all the dependence.

Cross-entropy measures the average logarithmic loss incurred when outcomes follow one probability distribution $P$, but a model assigns probabilities using another probability distribution $Q$. For a finite alphabet, write their probability mass functions as $p_i$ and $q_i$. With base-two logarithms,The expectation is under the true or stipulated source law $P$, rather than the model $Q$. Terms with $p_i=0$ contribute nothing. If $p_i>0$ but $q_i=0$ for any outcome, the cross-entropy is $+\infty$: the model rules out an event that can occur. The result is measured in bits; natural logarithms give nats.

The useful identity is cross-entropy equals entropy plus relative entropy:The source's information entropy $H(P)$ is the baseline uncertainty. The Kullback-Leibler divergence is the extra average cost of a mismatched model, with the true law first and model second. Thus optimizing $Q$ for a fixed $P$ reduces the mismatch, rather than changing the source's own uncertainty. For finite alphabets, the minimum is $H(P)$, attained exactly at $Q=P$.

Take a three-symbol source with $P=(1/2,1/4,1/4)$. A matched binary prefix code assigns $A\mapsto 0$, $B\mapsto 10$, $C\mapsto 11$, with lengths $(1,2,2)$. A mismatched prefix code instead assigns $A\mapsto 10$, $B\mapsto 11$, $C\mapsto 0$, with lengths $(2,2,1)$, corresponding to $Q=(1/4,1/4,1/2)$. Both prefix codes decode exactly, but the most common symbol is shorter in the matched code. Under the same source $P$,The mismatched code costs an extra (1/4) bit per source symbol on average. No information is lost; the penalty is inefficient representation of this source. The mismatch penalty in lossless coding explains when this coding interpretation is exact.

Cross-entropy is also useful when evaluating a predictive distribution. A report that places tiny probability on the realized outcome receives a large logarithmic loss, even when it usually predicts the most common outcome correctly. The loss convention is the negative of the reward convention in the logarithmic scoring rule. Its expected loss is uniquely minimized by the correct probability distribution, making it a strictly proper scoring rule. Strictly Proper Scoring Rules, Prediction, and Estimation by Tilmann Gneiting and others treats this connection in §3.1, Example 3; its discussion uses rewards, so the signs are reversed for the losses here.^[ref]

I. J. Good's Rational Decisions by I. J. Good appeared in 1952. Its published summary describes a method for encouraging accurate probability estimates and a connection with information theory. Strictly Proper Scoring Rules, Prediction, and Estimation by Tilmann Gneiting and others traces the logarithmic scoring rule at least as far back as this 1952 paper.^[ref]

For independent and identically distributed random variables with probability mass function $p$ on a finite alphabet, evaluate their realized strings under a fixed product model $q_n(x^n)={\prod }_{i}q(x_i)$. If $q$ is positive on the probability support of $p$, the strong law of large numbers givesalmost surely, using cross-entropy equals entropy plus relative entropy. If $q$ vanishes at an outcome of positive $p$ probability, that outcome eventually appears almost surely, making all subsequent product likelihoods zero and the limit $-\infty$. This matches infinite cross-entropy.

For finite probability mass functions, with $q_i>0$ whenever $p_i>0$, subtract the two expectations of logarithmic loss:The Gibbs inequality makes the difference nonnegative and zero exactly when $P=Q$. If $Q$ assigns zero probability to an outcome with positive probability under $P$, both the cross-entropy and the Kullback-Leibler divergence are infinite. The finite information entropy still leaves the displayed identity valid in the extended sense. In more general settings, undefined differences such as $\infty -\infty$ cannot be used as a derivation.

Suppose a binary prefix code has integer codeword lengths ${\ell }_i=-{\log }_2{q}_i$. Under a source with probability mass function $p_i$, its average codeword length is exactly cross-entropy, ${\sum }_i{p}_i{\ell }_i=H(P,Q)$. The Kraft inequality is satisfied because ${\sum }_i{2}^{-{\ell }_i}={\sum }_i{q}_i=1$. This exact interpretation requires suitable dyadic rational probabilities; a general $-{\log }_2{q}_i$ is not an integer codeword length. For positive $q_i$, a prefix code with lengths $\lceil -{\log }_2{q}_i\rceil$ exists by the Kraft inequality and has average length between $H(P,Q)$ and $H(P,Q)+1$. The ceiling function causes the one-symbol rounding overhead. The cross-entropy term describes mismatch overhead; those are distinct costs.

Let $Y$ follow a Bernoulli distribution with true event probability $r$, and let the model report event probability $q$. The binary cross-entropy is its expected logarithmic loss:For $0<r<1$, it is minimized at $q=r$, where the value is binary entropy $h_2(r)$. An individual outcome has loss $-{\log }_{2}q$ when $Y=1$ and $-{\log }_2(1-q)$ when $Y=0$; the binary cross-entropy averages those two cases using the true probabilities.

For $r=1/4$, truthful reporting gives $h_2(1/4)\approx 0.811278$ bits. Reporting $q=1/2$ gives one bit. Reporting $q=0.01$ givesBoth $q=1/4$ and $q=0.01$ select zero if one predicts only the more likely binary outcome. Their predicted probabilities are very different, however, and cross-entropy reveals the cost of underestimating the event that occurs a quarter of the time. Endpoint losses are computed using the realized outcome rather than multiplying zero by an undefined logarithm.

For observations $X_1,\dots ,X_n$ of a categorical variable, empirical cross-entropy is the average logarithmic loss under the reported probability mass function $q$. Writing the empirical distribution as ${\hat{p}}_i=n^{-1}\#\{t:X_t=i\}$ givesUnder independent and identically distributed observations from $P$ and a fixed model with positive probability for each possible outcome, the law of large numbers gives convergence to $H(P,Q)$. A finite empirical distribution is an estimate of $P$, rather than the true source law; unobserved outcomes need not be impossible. If the same data are used to choose $Q$, evaluating the resulting empirical cross-entropy on that data need not estimate its prediction performance on fresh observations without bias.

For a parametric statistical model with probability mass function $q_{\theta }$ and an independent and identically distributed model, minimizing empirical cross-entropy is maximum likelihood estimation. The logarithm is strictly increasing, so maximizing ${\prod }_t{q}_{\theta }(X_t)$ is equivalent to maximizing ${\sum }_t{\log }_2{q}_{\theta }(X_t)$. Negating and dividing by $n$ gives the displayed minimization. Restricting the model family can prevent the optimum from matching the empirical distribution, and a finite optimizing parameter need not exist. This algebra does not by itself establish accuracy on new data.^[ref]

If $U,V$ are independent random variables taking values in a countable additive group, conditioning on $V$ makes $U+V$ a translate of $U$. Consequently $H(U+V\mid V)=H(U)$, and conditional entropy gives $H(U+V)\ge H(U)$. If the two information entropies are finite, equality holds exactly when $U+V$ and $V$ are independent.

// Target: probability-and-statistics.bigb

This is the entropy in base $q$ of the distribution with one symbol of probability $1-x$ and each of the other $q-1$ symbols of probability $x/(q-1)$. It increases on $[0,1-1/q]$ from zero to one, and governs the exponential volume of a Hamming ball. For $q=2$ it is the binary entropy function.

// Target: algebra.bigb

The self-information of a positive-probability outcome is the negative logarithm of its probability. Independent outcomes have additive self-information. Its expectation is Shannon entropy.

// Target: probability-and-statistics.bigb

For a vertex law $P$ on a graph, choose $X\sim P$ and an independent set valued random variable $Y$ containing $X$ almost surely. Minimize their mutual information over all such joint laws. This defines graph entropy. It differs from the entropy of a proper coloring, because the random independent set need not be a deterministic color class.

If one probability in a $k$-point distribution equals $t$, normalize the remaining $k-1$ probabilities to $q$. Then $H=h(t)+(1-t)H(q)\le h(t)+(1-t){\log }_2(k-1)$. Equality for $t<1$ means that the remaining probabilities are equal. This combines the binary entropy of the distinguished event with the largest residual Shannon entropy.

For a finite-valued random variable, an epsilon-sufficient set $S$ retains probability at least $1-ϵ$. This notion is about covering probability mass and is unrelated to a sufficient statistic. For an IID random variable source, any such set of words with fixed $ϵ<1$ requires exponentially many words at a rate approaching the Shannon entropy.

For finite-alphabet independent and identically distributed random variables, fixed $0\le ϵ<1$ and $\delta >0$, every epsilon-sufficient set $S_n$ of length-$n$ words satisfies $|S_n|\ge (1-ϵ)2^{n(H-\delta )}/2$ for sufficiently large $n$. Intersect $S_n$ with a typical set whose excluded mass is at most $(1-ϵ)/2$. The intersection retains that much probability and each word has probability at most $2^{-n(H-\delta )}$.

Differential entropy describes the spread of a continuous probability distribution relative to the volume scale of its coordinates. For a real random variable $X$ with probability density function $f$, use natural logarithms and writeThe convention is $0\ln 0=0$. For a finite value, require $\int f|\ln f|<\infty$; if both positive and negative contributions are infinite, the expression is undefined. Unlike a probability, a probability density can exceed one. Consequently differential entropy can be negative. Base-two logarithms give $h_2=h_e/\ln 2$, in bits instead of nats.^[ref]

This quantity does not give the exact number of bits needed to store a real number. A recording device supplies a resolution; equal-width bin entropy connects that resolution to discrete information entropy. The useful continuous quantity is what remains after removing the increasing cost of specifying finer bins. Its dependence on coordinates is visible already in differential entropy of a uniform law.

For a continuous uniform distribution on an interval of length $L>0$, the probability density is $1/L$. Direct integration givesThus a uniform law on an interval of length $1/2$ has differential entropy $-\ln 2$, while length two gives $\ln 2$. Both laws have total probability one. The shorter interval has larger probability density, making $-\ln f(X)$ negative; this is neither a negative probability nor a negative discrete information entropy.

For a normal distribution with mean $m$ and positive variance ${\sigma }^2$, expand the logarithm of its probability density function:The second term has expected value $1/2$, hence $h_e(X)=\frac{1}{2}\ln (2\pi e{\sigma }^2)$. Translation of the mean has no effect, while shrinking the standard deviation lowers differential entropy. A standard normal distribution has approximately $1.41894$ nats. This connects the continuous spread calculation with entropy power, whose normalization matches a Gaussian distribution's variance.

Let $X$ have a probability density function on an open interval, and let $g$ be continuously differentiable, strictly increasing there, with $g^{\prime }(x)>0$. The change of variables formula givesIf the differential entropy and the expected absolute logarithm of $g^{\prime }(X)$ are finite, taking the expectation of the negative logarithm proves the displayed formula, for a fixed base $b>1$. Local stretching reduces probability density; its contribution to differential entropy is the expected logarithmic stretch. The map is invertible, so this change does not mean that the transformation destroyed information about $X$.

Take $X$ with continuous uniform distribution on $(0,1)$ and set $Y=X^2$. This is an increasing invertible transformation on that interval. Its cumulative distribution function is $F_Y(y)=\sqrt{y}$, givingAlthough the probability density diverges near zero, it is integrable. The logarithmic stretch is integrable too:The inverse function $X=\sqrt{Y}$ recovers every input in the interval. The changed differential entropy records a changed coordinate volume, rather than a failure of recovery.

// Target: probability-and-statistics.bigb

For real $a\ne 0$ and $b$, the change of variables formula gives $f_{aX+b}(ax+b)=f_X(x)/|a|$. ThereforeThe absolute value also covers a decreasing transformation. If a numerical length changes from metres to centimetres, $a=100$ and the differential entropy changes by $\ln 100\approx 4.60517$ nats, although the physical uncertainty is the same. The reference coordinate scale must therefore accompany an absolute differential entropy value.

Partition $(0,1)$ into $n$ bins of width $\delta =1/n$, and let the discrete random variable $Q_n$ record the bin containing $X$. This is a quantization of $X$. Its probability mass function isand its discrete information entropy is $H_e(Q_n)=-{\sum }_j{p}_j\ln p_j$. Let ${\bar{f}}_j=p_j/\delta$ be the average probability density within bin $j$. Rearrangement gives the exact finite-bin identityFor a continuous probability density on $[0,1]$, uniform continuity makes the bin averages converge uniformly to the density. Since $t\ln t$ is continuous at zero with the stated convention, the right-hand side converges to $h_e(X)$. Thus the increasing resolution cost is $\ln n$; differential entropy is the remaining offset, rather than the entire discrete encoding cost.

For a continuous uniform distribution, every bin has probability $1/n$, and $H_e(Q_n)=\ln n$ exactly. For the continuous uniform distribution's squared transformation above, the exact bin probabilities areThe diagram computes these masses directly, including the bin next to the integrable singularity. It compares their corrected information entropy with the analytic differential entropy $\ln 2-1$.

Apply the same invertible differentiable transformation to two probability distributions with densities $f$ and $r$. Both transformed densities acquire the same Jacobian determinant factor. Write their transformed densities as $f_g$ and $r_g$. Their ratio is unchanged, and the integration measure transforms with the numerator density. Consequently their relative entropy is unchanged:The densities describe two alternative laws in the same coordinate, with a nonzero Jacobian determinant on their common domain, and the expression is assumed well defined. This explains why comparing two laws can survive a change of measurement units even though either law's differential entropy changes. Mutual information, formulated as relative entropy from a product of marginals, has the same invariance under invertible changes to either observed coordinate.

Differential entropy is also the entropy term in the evidence lower bound for continuous variational inference. Its coordinate dependence must be combined consistently with that of the model density; changing units does not justify changing only one term of the objective.

In A Mathematical Theory of Communication by Claude Shannon (1948), section 20 extends the discrete entropy formula to continuous densities. Properties 8–9 explicitly discuss coordinate dependence, possible negative values and linear rescaling. Claude Shannon relates this coordinate convention to the engineering use of entropy differences, rather than treating an isolated continuous entropy as an invariant count of messages.^[ref]

The entropy power of an $n$-dimensional random vector with finite differential entropy in bits is the variance per coordinate of an isotropic Gaussian distribution with the same differential entropy. The normalization makes $N(X)={\sigma }^2$ when $X$ has covariance matrix ${\sigma }^2{I}_n$.

// Target: probability-and-statistics.bigb

For independent $n$-dimensional random vectors with densities and finite differential entropies, the entropy power inequality statesThe sum's differential entropy is required to be defined; a value $+\infty$ makes the inequality immediate. Equality for finite nondegenerate densities occurs for Gaussian distributions with proportional covariance matrices.

// Target: probability-and-statistics.bigb

For independent positive random variables $Y_1,Y_2$ with densities, finite differential entropies and finite ${\mu }_i=\mathbb{E}{\log }_2{Y}_i$, applying the entropy power inequality to $Z_i=\ln Y_i$ givesIndeed $h(Z_i)=h(Y_i)-{\mu }_i$, while $h(Y_1{Y}_2)=h(Z_1+Z_2)+{\mu }_1+{\mu }_2$, provided the sum's differential entropy is defined. Positivity alone does not ensure these expressions exist: $Y=e^T$ with $T$ having Cauchy distribution has undefined $\mathbb{E}{\log }_{2}Y$.

// Target: algebra.bigb

For a discrete random variable $X$ with a finite or countable alphabet and probability mass function $p_x=\mathbb{P}(X=x)$, define $p_{\max }={\max }_x{p}_x$. Its min-entropy, using base-two logarithms, isThe unit is the bit. This is the smallest information content of any possible outcome: it records the easiest outcome to predict, whereas Shannon entropy averages information content over all outcomes. A normalized countable probability mass function has an attained positive maximum, so $H_{\infty }(X)$ is finite even when Shannon entropy is infinite. Outcomes of zero probability cannot minimize the surprise and are omitted from the minimum; including extra impossible symbols changes neither quantity.^[ref]

Suppose someone knows the probability mass function but receives no observation correlated with $X$, and must name exactly one outcome. Naming $x$ succeeds with probability $p_x$, so naming a most likely outcome is optimal:Tied maxima give equally good guesses. Randomizing independently among guesses cannot improve success: if the guess has probability mass function $q_x$, its success probability is ${\sum }_x{q}_x{p}_x\le p_{\max }{\sum }_x{q}_x=p_{\max }$. This is the simplest instance of maximum a posteriori decoding. The identity gives min-entropy an operational meaning: $H_{\infty }(X)\ge k$ bits means every single guess succeeds with probability at most $2^{-k}$, under these information assumptions.^[ref]

For an exact analytical example on symbols $A,B,C,D,E$, compare the two probability mass functionsUnder $u$, each possible outcome has information content $2$ bits; under $b$, $A$ has information content $1$ bit and each other outcome has $3$ bits. Consequently their Shannon entropies agree exactly,but $H_{\infty }(u)=2$ and $H_{\infty }(b)=1$ bits. A best single guess therefore succeeds with probability $1/4$ under $u$ and $1/2$ under $b$. The same average surprise can hide a factor of two in immediate predictability. The zero entry in $u$ contributes zero to Shannon entropy, using $0{\log }_{2}0=0$.

For the discrete uniform distribution on $m$ possible outcomes, $H_{\infty }={\log }_{2}m$ bits and the best single guess succeeds with probability $1/m$. A certain outcome has $H_{\infty }=0$ and success probability one. Thus min-entropy is useful when a common outcome could defeat an attempt to make outputs unpredictable, even if rare alternatives raise Shannon entropy.

For cryptography, this unconditional calculation alone does not certify security: an observer might know $X$ already. If a report $Y=X$ reveals a uniform four-outcome source, its unconditional min-entropy remains $2$ bits but guessing after the report succeeds with probability one. For finite classical reports, the appropriate success calculation uses conditional probabilities and maximum a posteriori decoding:The conditional entropy $H(X\mid Y)$ instead averages remaining surprise. A security argument must specify the observer's information and the protocol's required guarantee; neither unconditional min-entropy nor Shannon entropy supplies those assumptions.

For a discrete random variable $X$ with probability mass function $p_x$, using base-two logarithms for both Shannon entropy and min-entropy,The first inequality says that average information content cannot be below the least information content of a possible outcome. For a finite alphabet, equality $H(X)=H_{\infty }(X)$ holds exactly when the probability mass function is uniform on its positive-probability outcomes. Impossible outcomes do not need to share that positive probability. In particular, $(1/2,1/2,0)$ has both quantities equal to $1$ bit, while $(1,0,0)$ has both zero. The same equality characterization holds for a countable alphabet: equality forces only finitely many positive-probability outcomes, because all must have the same positive mass. The uniform maximum of Shannon entropy is discussed in Shannon's original treatment.^[ref]

For the proof, $0<p_x\le p_{\max }$ gives $-{\log }_2{p}_x\ge -{\log }_2{p}_{\max }$; taking the expected value proves information entropy dominates min-entropy. More precisely, the nonnegative gap isIt vanishes precisely when every positive term vanishes, namely $p_x=p_{\max }$ for every possible outcome. With $m$ such outcomes, normalization forces $p_{\max }=1/m$. If Shannon entropy is infinite, the inequality still holds and is strict, because min-entropy of a countable probability mass function is finite. No subtraction of two infinite quantities is needed.

An analytical family makes the gap visible. Give $A$ probability $q$ and each of $B,C,D,E$ probability $(1-q)/4$, with $1/5\le q\le 1$. Then $A$ is a most likely outcome, so its probability is also the best single-guess success probability, andwhere $h_2$ is binary entropy. The entropy bound with one prescribed probability explains the second expression: it separates whether $A$ occurred from which of four equally likely alternatives occurred. At $q=1/5$, all five outcomes are uniform and both quantities are ${\log }_{2}5\approx 2.3219$ bits. At $q=1/2$, Shannon entropy is $2$ bits while min-entropy is $1$ bit, reproducing the biased example in min-entropy. At $q=1$, both are zero; the four impossible alternatives contribute nothing. For every $1/5<q<1$, the probability mass function is nonuniform on its possible outcomes and the gap is strictly positive.

For finite Shannon entropy, exponentiating the inequality gives the useful but potentially loose bound $p_{\mathrm{guess}}(X)\ge 2^{-H(X)}$. Equality holds exactly for a discrete uniform distribution on the possible outcomes. Thus Shannon entropy alone supplies a lower bound on optimal success, not the upper bound needed to promise that every guess is unlikely. Shannon's source coding theorem concerns average compression under its hypotheses, while min-entropy answers the single-guess question directly. The two quantities serve different tasks even when both are expressed in bits.

The varentropy of a discrete random variable with mass function $P$ is the variance of its information content:

The binary entropy function is the information entropy of a Bernoulli random variable:with endpoint values zero. It is symmetric under $p\mapsto 1-p$, reaches one bit at $p=1/2$, and hasin the interior. Its strict concavity expresses why an even mixture of two outcomes has more uncertainty than a strong bias toward either one. The curve in mutual information also shows how this same uncertainty becomes the loss term in binary symmetric channel capacity.

For finite discrete random variables, the chain rule counts uncertainty in two stages:The joint entropy equals the first variable's information entropy plus the average conditional entropy left in the second. All logarithms here have base two, so the unit is the bit; zero-probability terms contribute zero. The identity holds without assuming independence.^[ref]

Iterating gives $H(X_1,\dots ,X_n)={\sum }_{i}H(X_i\mid X_1,\dots ,X_{i-1})$. The first summand means $H(X_1)$, with no earlier observations. Each later summand counts only the new uncertainty after the preceding random variables are known. For independent random variables, every term is its ordinary information entropy; for dependent ones, adding those ordinary information entropies can count shared uncertainty repeatedly. This is the reason the chain rule for information entropy is useful in lossless source coding and in calculating an entropy rate.

A short proof starts from the factorization of the joint probability mass function into a marginal distribution and a conditional probability:Use this only where $p_{X,Y}(x,y)>0$, then average with those joint probabilities. Summing over $y$ in the first term gives $H(X)$; the second is precisely $H(Y\mid X)$. Repeating the same step with a longer random vector proves the iterated identity. Reversing the order also gives $H(X,Y)=H(Y)+H(X\mid Y)$: the two contributions can change with the order, but their total cannot.

Let $A$ be a fair bit, let $N$ be an independent Bernoulli random variable with $\mathbb{P}(N=1)=p$, and set $B=A\oplus N$, where $\oplus$ is exclusive or. Thus $N=1$ flips the first bit. This constructs a binary symmetric channel with a fair input, for $0\le p\le 1$. The joint probability mass function isThe column sums show that $B$ is also a fair bit, whatever $p$ is. Both marginal distributions therefore have information entropy one bit. Given $A$, only the flip decision remains unknown, giving conditional entropy $H(B\mid A)=h_2(p)$, whereis the binary entropy, with $h_2(0)=h_2(1)=0$. The chain rule for information entropy and the definition of mutual information now give

Take $p=1/10$. The four joint probabilities, in row order, are $(0.45,0.05,0.05,0.45)$. Directly from the definition of joint entropy,Conditioning on either value of $A$ gives probabilities $0.9$ and $0.1$ for the two values of $B$, henceAdding $H(A)+H(B)=2$ would overcount by exactly $I(A;B)$. These are average uncertainties under the stipulated probability distribution; the four possible pairs are still represented by four distinct outcomes.

There is another way to see the decomposition: $(A,B)\mapsto (A,A\oplus B)=(A,N)$ is a bijection. It merely relabels the four outcomes, so it preserves joint entropy. The independence of $A$ and $N$ then gives $H(A,B)=H(A,N)=H(A)+H(N)=1+h_2(p)$. Describing the initial bit and the flip decision separates shared uncertainty from new uncertainty.

At $p=0$, the bits are identical: $B=A$, so $H(B\mid A)=0$ and $I(A;B)=1$. At $p=1$, they are complementary: $B=1-A$, with exactly the same conditional entropy and mutual information. In either case, observing one determines the other. At $p=1/2$, all four joint probabilities are $1/4$; the bits are independent, $H(A,B)=2$, and $I(A;B)=0$. Perfect disagreement is therefore just as informative as perfect agreement.

For $p>1/2$, guessing $A=1-B$ succeeds with probability $p$, better than guessing $A=B$. Changing $B$ to $1-B$ replaces $p$ by $1-p$ and preserves mutual information, since the relabeling is invertible. What matters is how well the observation distinguishes the input, rather than how often its symbol matches the input. The same symmetry makes the binary entropy curve peak at independence, while the mutual information curve reaches its minimum there.