Cosma Rohilla Shalizi Created 2026-10-14 Updated 2026-11-05
Author of teaching material in statistics, data analysis and information theory. His explanations connect probabilistic models, inference and predictive checks rather than treating a fitted numerical result as its own validation.[ref]
Cross-entropy 2026-10-14
Cross-entropy measures the average logarithmic loss incurred when outcomes follow one probability distribution , but a model assigns probabilities using another probability distribution . For a finite alphabet, write their probability mass functions as and . With base-two logarithms,
The expectation is under the true or stipulated source law , rather than the model . Terms with contribute nothing. If but for any outcome, the cross-entropy is : the model rules out an event that can occur. The result is measured in bits; natural logarithms give nats.
The useful identity is cross-entropy equals entropy plus relative entropy:
The source's information entropy is the baseline uncertainty. The Kullback-Leibler divergence is the extra average cost of a mismatched model, with the true law first and model second. Thus optimizing for a fixed reduces the mismatch, rather than changing the source's own uncertainty. For finite alphabets, the minimum is , attained exactly at .
Take a three-symbol source with . A matched binary prefix code assigns , , , with lengths . A mismatched prefix code instead assigns , , , with lengths , corresponding to . Both prefix codes decode exactly, but the most common symbol is shorter in the matched code. Under the same source ,
The mismatched code costs an extra (1/4) bit per source symbol on average. No information is lost; the penalty is inefficient representation of this source. The mismatch penalty in lossless coding explains when this coding interpretation is exact.
Figure 1.
Two lossless prefix codes have different average lengths for the same source
. Both sets of codewords are prefix codes. The source probabilities are fixed at . Each colored contribution is source probability times codeword length; changing which symbol receives the shortest word increases the total from to bits per source symbol. Details.
Cross-entropy is also useful when evaluating a predictive distribution. A report that places tiny probability on the realized outcome receives a large logarithmic loss, even when it usually predicts the most common outcome correctly. The loss convention is the negative of the reward convention in the logarithmic scoring rule. Its expected loss is uniquely minimized by the correct probability distribution, making it a strictly proper scoring rule. Strictly Proper Scoring Rules, Prediction, and Estimation by Tilmann Gneiting and others treats this connection in §3.1, Example 3; its discussion uses rewards, so the signs are reversed for the losses here.[ref]
I. J. Good's Rational Decisions by I. J. Good appeared in 1952. Its published summary describes a method for encouraging accurate probability estimates and a connection with information theory. Strictly Proper Scoring Rules, Prediction, and Estimation by Tilmann Gneiting and others traces the logarithmic scoring rule at least as far back as this 1952 paper.[ref]
George Armitage Miller Created 2026-10-25 Updated 2026-11-05
A cognitive psychologist who applied information theory to limits on human information processing. His 1956 paper distinguished immediate human memory from judgments of isolated sensory magnitudes and emphasized that chunking changes how much information a remembered unit contains. Familiar groups of items can be retained as units, so a count of remembered units is not a universal limit on bits of information. [ref]
Figure 1.
George Armitage Miller speaking at the first Association for Psychological Science convention in 1989
. Photographer unknown; originally published by the Association for Psychological Science. Historical identification photograph hosted by English Wikipedia as non-free media. Source.
Information entropy Created 2026-09-24 Updated 2026-10-14
For a finite discrete random variable with probability mass function , its information entropy is the expected value of its information content:
Base-two logarithms give bits; choosing a different base rescales the units. The convention is justified by a limit, so outcomes of zero probability contribute nothing. The definition also applies to countable distributions, but the sum may be infinite.
A fair bit has . A certain outcome has . A Bernoulli random variable that is one with probability has bits, because most of its outcomes are predictable. A rare outcome still has large information content: the outcome of probability contributes bits of surprise when it occurs. Entropy is the average surprise, rather than the surprise of each outcome.
For possible outcomes, , with the upper bound attained exactly by the uniform distribution on a finite set. The Gibbs inequality proves this by comparing with the uniform distribution: . Thus increasing predictability reduces the required average description length. The four-symbol prefix code in information theory attains bits exactly; for general probabilities, integer codeword lengths prevent such a perfect match for every individual symbol. Shannon's source coding theorem explains how coding sufficiently long independent blocks approaches the entropy per symbol.
The logarithm is especially natural because independent outcomes have additive surprise: . Taking expected values gives additive information entropy for independent random variables; for dependent ones, the chain rule for information entropy replaces the second term by conditional entropy. This is why the same quantity can govern both a single observation and a long message.
Basic-option schedule. The central description gives indicative choices including algebraic topology, category theory, information theory and quantum processes and computation; annual availability follows the handbook. 2026–27 handbook. Advanced-option schedule. Indicative choices include algebraic geometry, automata, logic and games, and axiomatic set theory. At least five lecture courses are required overall. Compulsory dissertation on a topic related to the taught material, undertaken in Trinity term and the summer.
Paul C. Shields Created 2026-10-13 Updated 2026-11-05
Coauthor with Imre Csiszár of a tutorial connecting information theory with statistics. It develops information measures as tools for statistical modeling and inference.[ref]
Rational Decisions by I. J. Good Created 2026-10-13 Updated 2026-11-05
Published in the Journal of the Royal Statistical Society, Series B, volume 14, number 1, pp. 107–114.
Cited passage: the published summary describes encouraging accurate probability estimates and a connection with information theory.
Satoshi Watanabe (physicist) Created 2026-10-13 Updated 2026-11-05
Satoshi Watanabe analyzed dependence among many random variables using information theory. The total correlation compares the sum of individual Shannon entropy values with the joint entropy, measuring information stored in their dependence: independent variables give zero difference, while correlated variables can share information. The multivariable construction goes beyond checking one pair at a time, helping distinguish a collective statistical structure from a collection of unrelated observations.[ref]
Figure 1.
Portrait of Satoshi Watanabe
. Asahi Shimbun; Asahi Graph, 4 May 1949. Public domain in Japan and United States, as file states (old Japanese photograph; copyright expired before 1970; no URAA restoration). Source and usage information on the linked Wikimedia file page. Source.