Proper scoring rule (source code)

= Proper scoring rule
{title2=$\mathbb E_QS(Q,X)\ge\mathbb E_QS(P,X)$}

= Propriety
{synonym}

A <proper scoring rule> makes reporting the true <probability distribution> optimal in <expected value>, for every true <probability distribution> in the allowed class of forecasts. In the reward convention, the inequality in the title holds for every report $P$ and true law $Q$. The <expected value> is always taken under $Q$, because the report does not change how the outcome is generated. In the loss convention the same definition reads
$$
\mathbb E_Q[L(Q,X)]\leq\mathbb E_Q[L(P,X)]\qquad\text{for every admissible }P,Q.
$$
This requirement makes a <proper scoring rule> useful for eliciting <probabilities> and comparing <predictive distributions>: deliberately distorting a known forecast cannot improve its expected reward or expected loss. The reward convention and its strict version are set out in <Strictly Proper Scoring Rules, Prediction, and Estimation by Tilmann Gneiting and others> (Section 1).

For the <Bernoulli distribution> with $q=0.7$, the <Brier score> has expected loss $(p-0.7)^2+0.21$, and the negative <logarithmic scoring rule> has expected loss $-0.7\ln p-0.3\ln(1-p)$. Both are minimized at the honest report $p=0.7$. The <linear probability score>, by contrast, maximizes its expected reward at $p=1$. These are statements about averaging over both possible outcomes with their actual <probabilities>.

\Image[/proper-score-expected-losses.png]
{title=Expected losses of the <Brier score> and <logarithmic scoring rule> favor $p=q=0.7$; the <linear probability score> rewards $p=1$}

The left panel displays each excess expected loss divided by that rule's expected loss at $p=q$: $0.21$ for the <Brier score> and approximately $0.610864$ for the <logarithmic scoring rule>. This normalization permits comparison of curve shapes and preserves the minimizing report, since it subtracts a constant and divides by a positive constant independent of $p$. Zero means no excess expected loss. The right panel shows the unmodified <linear probability score> reward, with its maximizing direction labeled explicitly.