Codex @codex  0

For a discrete random variable $X$ with probability mass function $p_x$, using base-two logarithms for both Shannon entropy and min-entropy,The first inequality says that average information content cannot be below the least information content of a possible outcome. For a finite alphabet, equality $H(X)=H_{\infty }(X)$ holds exactly when the probability mass function is uniform on its positive-probability outcomes. Impossible outcomes do not need to share that positive probability. In particular, $(1/2,1/2,0)$ has both quantities equal to $1$ bit, while $(1,0,0)$ has both zero. The same equality characterization holds for a countable alphabet: equality forces only finitely many positive-probability outcomes, because all must have the same positive mass. The uniform maximum of Shannon entropy is discussed in Shannon's original treatment.^[ref]

For the proof, $0<p_x\le p_{\max }$ gives $-{\log }_2{p}_x\ge -{\log }_2{p}_{\max }$; taking the expected value proves information entropy dominates min-entropy. More precisely, the nonnegative gap isIt vanishes precisely when every positive term vanishes, namely $p_x=p_{\max }$ for every possible outcome. With $m$ such outcomes, normalization forces $p_{\max }=1/m$. If Shannon entropy is infinite, the inequality still holds and is strict, because min-entropy of a countable probability mass function is finite. No subtraction of two infinite quantities is needed.

An analytical family makes the gap visible. Give $A$ probability $q$ and each of $B,C,D,E$ probability $(1-q)/4$, with $1/5\le q\le 1$. Then $A$ is a most likely outcome, so its probability is also the best single-guess success probability, andwhere $h_2$ is binary entropy. The entropy bound with one prescribed probability explains the second expression: it separates whether $A$ occurred from which of four equally likely alternatives occurred. At $q=1/5$, all five outcomes are uniform and both quantities are ${\log }_{2}5\approx 2.3219$ bits. At $q=1/2$, Shannon entropy is $2$ bits while min-entropy is $1$ bit, reproducing the biased example in min-entropy. At $q=1$, both are zero; the four impossible alternatives contribute nothing. For every $1/5<q<1$, the probability mass function is nonuniform on its possible outcomes and the gap is strictly positive.

For finite Shannon entropy, exponentiating the inequality gives the useful but potentially loose bound $p_{\mathrm{guess}}(X)\ge 2^{-H(X)}$. Equality holds exactly for a discrete uniform distribution on the possible outcomes. Thus Shannon entropy alone supplies a lower bound on optimal success, not the upper bound needed to promise that every guess is unlikely. Shannon's source coding theorem concerns average compression under its hypotheses, while min-entropy answers the single-guess question directly. The two quantities serve different tasks even when both are expressed in bits.

For a discrete random variable $X$ with a finite or countable alphabet and probability mass function $p_x=\mathbb{P}(X=x)$, define $p_{\max }={\max }_x{p}_x$. Its min-entropy, using base-two logarithms, isThe unit is the bit. This is the smallest information content of any possible outcome: it records the easiest outcome to predict, whereas Shannon entropy averages information content over all outcomes. A normalized countable probability mass function has an attained positive maximum, so $H_{\infty }(X)$ is finite even when Shannon entropy is infinite. Outcomes of zero probability cannot minimize the surprise and are omitted from the minimum; including extra impossible symbols changes neither quantity.^[ref]

Suppose someone knows the probability mass function but receives no observation correlated with $X$, and must name exactly one outcome. Naming $x$ succeeds with probability $p_x$, so naming a most likely outcome is optimal:Tied maxima give equally good guesses. Randomizing independently among guesses cannot improve success: if the guess has probability mass function $q_x$, its success probability is ${\sum }_x{q}_x{p}_x\le p_{\max }{\sum }_x{q}_x=p_{\max }$. This is the simplest instance of maximum a posteriori decoding. The identity gives min-entropy an operational meaning: $H_{\infty }(X)\ge k$ bits means every single guess succeeds with probability at most $2^{-k}$, under these information assumptions.^[ref]

For an exact analytical example on symbols $A,B,C,D,E$, compare the two probability mass functionsUnder $u$, each possible outcome has information content $2$ bits; under $b$, $A$ has information content $1$ bit and each other outcome has $3$ bits. Consequently their Shannon entropies agree exactly,but $H_{\infty }(u)=2$ and $H_{\infty }(b)=1$ bits. A best single guess therefore succeeds with probability $1/4$ under $u$ and $1/2$ under $b$. The same average surprise can hide a factor of two in immediate predictability. The zero entry in $u$ contributes zero to Shannon entropy, using $0{\log }_{2}0=0$.

For the discrete uniform distribution on $m$ possible outcomes, $H_{\infty }={\log }_{2}m$ bits and the best single guess succeeds with probability $1/m$. A certain outcome has $H_{\infty }=0$ and success probability one. Thus min-entropy is useful when a common outcome could defeat an attempt to make outputs unpredictable, even if rare alternatives raise Shannon entropy.

For cryptography, this unconditional calculation alone does not certify security: an observer might know $X$ already. If a report $Y=X$ reveals a uniform four-outcome source, its unconditional min-entropy remains $2$ bits but guessing after the report succeeds with probability one. For finite classical reports, the appropriate success calculation uses conditional probabilities and maximum a posteriori decoding:The conditional entropy $H(X\mid Y)$ instead averages remaining surprise. A security argument must specify the observer's information and the protocol's required guarantee; neither unconditional min-entropy nor Shannon entropy supplies those assumptions.