Adrian Raftery Created 2026-10-24 Updated 2026-11-05
Coauthor of research on proper scoring rules with Tilmann Gneiting and on Bayes factors with Robert E. Kass. These works concern evaluating probabilistic predictions and comparing statistical models.[ref]
Linear probability score Created 2026-10-07 Updated 2026-10-14
The linear probability score pays the reported probability of whichever binary outcome occurs. With report and observed , its reward is
It looks sensible: assign more probability to what happens and receive more reward. But the expected value under a Bernoulli distribution with true success probability is
This is an affine function of the report. When , the maximum over is at ; when , it is at . At , every report ties. Thus the linear probability score encourages reporting certainty about the more likely outcome and is not a proper scoring rule.
For the stipulated , the expected value of the reward is . Honest reporting earns in expectation, whereas earns . The forecaster improves expected reward by replacing a 70% forecast with a 100% forecast. The Brier score and logarithmic scoring rule avoid this incentive, as the expected-score diagram under proper scoring rule shows.
A scoring rule in the reward convention gives for announcing the probability distribution and then observing . Under genuine belief , the expected score is
A proper scoring rule satisfies for every admissible . A strictly proper scoring rule additionally requires equality only when the distributions coincide:
The expected scores must be well defined. A loss convention reverses the inequalities, but the question uses rewards.
Proper scoring rule Created 2026-10-07 Updated 2026-10-14
A proper scoring rule makes reporting the true probability distribution optimal in expected value, for every true probability distribution in the allowed class of forecasts. In the reward convention, the inequality in the title holds for every report and true law . The expected value is always taken under , because the report does not change how the outcome is generated. In the loss convention the same definition reads
This requirement makes a proper scoring rule useful for eliciting probabilities and comparing predictive distributions: deliberately distorting a known forecast cannot improve its expected reward or expected loss. The reward convention and its strict version are set out in Strictly Proper Scoring Rules, Prediction, and Estimation by Tilmann Gneiting and others (Section 1).
For the Bernoulli distribution with , the Brier score has expected loss , and the negative logarithmic scoring rule has expected loss . Both are minimized at the honest report . The linear probability score, by contrast, maximizes its expected reward at . These are statements about averaging over both possible outcomes with their actual probabilities.
The left panel displays each excess expected loss divided by that rule's expected loss at : for the Brier score and approximately for the logarithmic scoring rule. This normalization permits comparison of curve shapes and preserves the minimizing report, since it subtracts a constant and divides by a positive constant independent of . Zero means no excess expected loss. The right panel shows the unmodified linear probability score reward, with its maximizing direction labeled explicitly.
Scoring rule Created 2026-10-07 Updated 2026-10-14
A scoring rule turns a probability distribution forecast and an observed outcome into a number. For example, a forecast of rain with probability should express uncertainty more usefully than a bare prediction of rain. A scoring rule lets us compare such forecasts after observations arrive, and its choice determines whether exaggerating confidence pays. A proper scoring rule rewards an honest account of uncertainty in expected value.
There are two sign conventions. A reward is maximized; a loss function is minimized. The conversion reverses every comparison. Here the linear probability score is a reward, while the binary Brier score and the negative logarithmic scoring rule are losses. Calling all three a "score" without specifying the direction would hide the central difference.
Throughout the binary examples, the random variable has a Bernoulli distribution with actual success probability , and the report assigns probability to and to . The value is stipulated by this statistical model; it is not inferred from data or a claim that a forecasting system is calibrated. In practice the true predictive distribution is unknown, and the relevant is the conditional probability of success given the information available when the forecast is issued. Propriety evaluates incentives for reporting that predictive distribution; it does not supply the unknown probabilities.
Strictly proper scoring rule Created 2026-10-07 Updated 2026-10-14
A strictly proper scoring rule is a proper scoring rule for which the true probability distribution is the unique optimizer of the expected value. Equivalently, any different admissible probability distribution gives strictly larger expected loss. For probability density functions, two densities that agree almost everywhere represent the same probability distribution. Ordinary propriety permits ties: the constant reward is a proper scoring rule, because every report ties, but it is not a strictly proper scoring rule and elicits no information.
Strict propriety concerns expectation, not guaranteed success on each observation or each realized dataset. With the stipulated Bernoulli distribution , suppose the outcome is . The honest report then has Brier score loss , while has loss . The logarithmic loss similarly gives losses and . The honest forecast loses this comparison on that draw, which occurs with probability in the stipulated statistical model.
Figure 1.
Individual outcomes can favor a different report even though the expected Brier score and logarithmic loss are uniquely minimized at
. Details.
The loss curves favor , and the loss curves favor . Their weighted average, with weights and , favors . Choosing a report after observing would use information unavailable at forecast time. Even an average loss on a finite dataset can rank an honest forecast below a competing forecast through outcome variation; strict propriety guarantees the ranking in expected value under the specified information and true predictive distribution.
Tilmann Gneiting Created 2026-10-13 Updated 2026-11-05
Coauthor with Adrian Raftery of research on proper scoring rules for probabilistic forecasts. A proper rule rewards reporting the forecaster's actual probability distribution in expectation, connecting forecast evaluation with statistical estimation.[ref]