 Proper scoring rule

A proper scoring rule makes reporting the true probability distribution optimal in expected value, for every true probability distribution in the allowed class of forecasts. In the reward convention, the inequality in the title holds for every report $P$ and true law $Q$. The expected value is always taken under $Q$, because the report does not change how the outcome is generated. In the loss convention the same definition readsThis requirement makes a proper scoring rule useful for eliciting probabilities and comparing predictive distributions: deliberately distorting a known forecast cannot improve its expected reward or expected loss. The reward convention and its strict version are set out in Strictly Proper Scoring Rules, Prediction, and Estimation by Tilmann Gneiting and others (Section 1).

For the Bernoulli distribution with $q=0.7$, the Brier score has expected loss $(p-0.7{)}^2+0.21$, and the negative logarithmic scoring rule has expected loss $-0.7\ln p-0.3\ln (1-p)$. Both are minimized at the honest report $p=0.7$. The linear probability score, by contrast, maximizes its expected reward at $p=1$. These are statements about averaging over both possible outcomes with their actual probabilities.

The left panel displays each excess expected loss divided by that rule's expected loss at $p=q$: $0.21$ for the Brier score and approximately $0.610864$ for the logarithmic scoring rule. This normalization permits comparison of curve shapes and preserves the minimizing report, since it subtracts a constant and divides by a positive constant independent of $p$. Zero means no excess expected loss. The right panel shows the unmodified linear probability score reward, with its maximizing direction labeled explicitly.