= Scoring rule
{title2=$S(P,x)$}
A <scoring rule> turns a <probability distribution> forecast and an observed outcome into a number. For example, a forecast of rain with <probability> $0.7$ should express uncertainty more usefully than a bare prediction of rain. A <scoring rule> lets us compare such forecasts after observations arrive, and its choice determines whether exaggerating confidence pays. A <proper scoring rule> rewards an honest account of uncertainty in <expected value>.
There are two sign conventions. A reward $S(P,x)$ is maximized; a <loss function> $L(P,x)$ is minimized. The conversion $L=-S$ reverses every comparison. Here the <linear probability score> is a reward, while the binary <Brier score> and the negative <logarithmic scoring rule> are losses. Calling all three a "score" without specifying the direction would hide the central difference.
Throughout the binary examples, the <random variable> $Y$ has a <Bernoulli distribution> with actual success <probability> $q=0.7$, and the report $p\in[0,1]$ assigns <probability> $p$ to $Y=1$ and $1-p$ to $Y=0$. The value $q=0.7$ is stipulated by this <statistical model>; it is not inferred from data or a claim that a forecasting system is calibrated. In practice the true <predictive distribution> is unknown, and the relevant $q$ is the <conditional probability> of success given the information available when the forecast is issued. <Propriety> evaluates incentives for reporting that <predictive distribution>; it does not supply the unknown <probabilities>.
Back to article page
