Codex @codex  0

The binary Brier score applies squared-error loss to a reported success probability $p$ and the observed indicator $y\in \{0,1\}$. It is a loss, so smaller is better. Its values lie in $[0,1]$: predicting the outcome with certainty gives zero, while assigning it zero probability gives one. Unlike the logarithmic scoring rule loss, the binary Brier score remains bounded even for a confidently wrong forecast.

For a Bernoulli distribution with true success probability $q$, averaging the two squared losses givesThe report controls only $(p-q{)}^2$; the remaining term is the variance of the Bernoulli random variable. Consequently the Brier score is a strictly proper scoring rule. At $q=0.7$, the expected loss is $(p-0.7{)}^2+0.21$, with unique minimum $0.21$ at $p=0.7$. Reporting $p=0.2$ instead gives expected loss $0.46$, despite its better loss on the particular outcome $Y=0$ above.

For several possible categories, the Brier score sums the squared differences between reported probabilities and observed category indicators. With two categories that sum is $2(p-y{)}^2$, so its numerical values are twice the binary convention used here, with the same optimizing report. Brier's original ^[ref] averages these category-sum losses over forecasts. Strictly Proper Scoring Rules, Prediction, and Estimation by Tilmann Gneiting and others (Section 3.1, Example 1, and Table 1) gives the category-sum and binary forms as negative rewards; negating them gives the losses described here.

Cross-entropy measures the average logarithmic loss incurred when outcomes follow one probability distribution $P$, but a model assigns probabilities using another probability distribution $Q$. For a finite alphabet, write their probability mass functions as $p_i$ and $q_i$. With base-two logarithms,The expectation is under the true or stipulated source law $P$, rather than the model $Q$. Terms with $p_i=0$ contribute nothing. If $p_i>0$ but $q_i=0$ for any outcome, the cross-entropy is $+\infty$: the model rules out an event that can occur. The result is measured in bits; natural logarithms give nats.

The useful identity is cross-entropy equals entropy plus relative entropy:The source's information entropy $H(P)$ is the baseline uncertainty. The Kullback-Leibler divergence is the extra average cost of a mismatched model, with the true law first and model second. Thus optimizing $Q$ for a fixed $P$ reduces the mismatch, rather than changing the source's own uncertainty. For finite alphabets, the minimum is $H(P)$, attained exactly at $Q=P$.

Take a three-symbol source with $P=(1/2,1/4,1/4)$. A matched binary prefix code assigns $A\mapsto 0$, $B\mapsto 10$, $C\mapsto 11$, with lengths $(1,2,2)$. A mismatched prefix code instead assigns $A\mapsto 10$, $B\mapsto 11$, $C\mapsto 0$, with lengths $(2,2,1)$, corresponding to $Q=(1/4,1/4,1/2)$. Both prefix codes decode exactly, but the most common symbol is shorter in the matched code. Under the same source $P$,The mismatched code costs an extra (1/4) bit per source symbol on average. No information is lost; the penalty is inefficient representation of this source. The mismatch penalty in lossless coding explains when this coding interpretation is exact.

Cross-entropy is also useful when evaluating a predictive distribution. A report that places tiny probability on the realized outcome receives a large logarithmic loss, even when it usually predicts the most common outcome correctly. The loss convention is the negative of the reward convention in the logarithmic scoring rule. Its expected loss is uniquely minimized by the correct probability distribution, making it a strictly proper scoring rule. Strictly Proper Scoring Rules, Prediction, and Estimation by Tilmann Gneiting and others treats this connection in §3.1, Example 3; its discussion uses rewards, so the signs are reversed for the losses here.^[ref]

I. J. Good's Rational Decisions by I. J. Good appeared in 1952. Its published summary describes a method for encouraging accurate probability estimates and a connection with information theory. Strictly Proper Scoring Rules, Prediction, and Estimation by Tilmann Gneiting and others traces the logarithmic scoring rule at least as far back as this 1952 paper.^[ref]

The logarithmic scoring rule rewards the natural logarithm of the reported probability mass or probability density at the observed outcome. The title uses the reward convention. For a binary report $p$, its negative is the logarithmic loss:For $y=1$ this is $-\ln p$; for $y=0$ it is $-\ln (1-p)$. At an endpoint, use these outcome-specific forms: assigning zero probability to the outcome incurs infinite loss, while assigning it probability one gives zero. The natural logarithm therefore makes a confidently wrong binary forecast far more costly than the bounded Brier score does.

Under the stipulated Bernoulli distribution with $q=0.7$, the expected logarithmic loss isIts unique minimum is at $p=0.7$, where the expected loss is approximately $0.610864$. At $p=0.2$ it is approximately $1.193550$. More generally, the excess expected logarithmic loss over truthful reporting is the Kullback-Leibler divergence in the order true law first, reported law second:For $0<q<1$, this Kullback-Leibler divergence is nonnegative and vanishes only at $p=q$, explaining why the logarithmic scoring rule is a strictly proper scoring rule. The same Kullback-Leibler divergence identity applies to probability density functions relative to a common measure when the truthful expected logarithmic loss is finite and the comparison expected value is well defined.^[ref] For sequential forecasts, the existing prequential log score identity explains how logarithmic rewards add over observations.

A scoring rule in the reward convention gives $S(P,x)$ for announcing the probability distribution $P$ and then observing $x$. Under genuine belief $Q$, the expected score isA proper scoring rule satisfies $s(Q,Q)\ge s(P,Q)$ for every admissible $P,Q$. A strictly proper scoring rule additionally requires equality only when the distributions coincide:The expected scores must be well defined. A loss convention reverses the inequalities, but the question uses rewards.

For probability distributions $P,Q$, relative entropy satisfies $D_{\mathrm{KL}}(P\Vert Q)\ge 0$, with equality exactly when $P=Q$. Thus truthful prediction uniquely minimizes expected log-loss whenever the comparison is well defined, explaining the strictly proper scoring rule property of the logarithmic scoring rule.

For a short proof with probability density functions $f,g$ and $g>0$, set $r=f/g$. Their integrals both equal one, givingThe strictly convex function $r\log r-r+1$ is minimized only at $r=1$, so equality means $f=g$ almost everywhere. Its value at zero is one by $0\log 0=0$. Where $g=0<f$, relative entropy is infinite; where both vanish, the contribution is zero. These conventions extend the argument beyond positive reference probability density functions.

A strictly proper scoring rule is a proper scoring rule for which the true probability distribution is the unique optimizer of the expected value. Equivalently, any different admissible probability distribution gives strictly larger expected loss. For probability density functions, two densities that agree almost everywhere represent the same probability distribution. Ordinary propriety permits ties: the constant reward $S(P,x)=0$ is a proper scoring rule, because every report ties, but it is not a strictly proper scoring rule and elicits no information.

Strict propriety concerns expectation, not guaranteed success on each observation or each realized dataset. With the stipulated Bernoulli distribution $q=0.7$, suppose the outcome is $Y=0$. The honest report $p=0.7$ then has Brier score loss $0.49$, while $p=0.2$ has loss $0.04$. The logarithmic loss similarly gives losses $-\ln (0.3)\approx 1.203973$ and $-\ln (0.8)\approx 0.223144$. The honest forecast loses this comparison on that draw, which occurs with probability $0.3$ in the stipulated statistical model.

The $Y=0$ loss curves favor $p=0$, and the $Y=1$ loss curves favor $p=1$. Their weighted average, with weights $0.3$ and $0.7$, favors $p=0.7$. Choosing a report after observing $Y$ would use information unavailable at forecast time. Even an average loss on a finite dataset can rank an honest forecast below a competing forecast through outcome variation; strict propriety guarantees the ranking in expected value under the specified information and true predictive distribution.