Strictly proper scoring rule (source code)

= Strictly proper scoring rule

= Strict propriety
{synonym}

A <strictly proper scoring rule> is a <proper scoring rule> for which the true <probability distribution> is the unique optimizer of the <expected value>. Equivalently, any different admissible <probability distribution> gives strictly larger expected loss. For <probability density functions>, two densities that agree <almost everywhere> represent the same <probability distribution>. Ordinary <propriety> permits ties: the constant reward $S(P,x)=0$ is a <proper scoring rule>, because every report ties, but it is not a <strictly proper scoring rule> and elicits no information.

<Strict propriety> concerns <expectation>, not guaranteed success on each observation or each realized dataset. With the stipulated <Bernoulli distribution> $q=0.7$, suppose the outcome is $Y=0$. The honest report $p=0.7$ then has <Brier score> loss $0.49$, while $p=0.2$ has loss $0.04$. The <logarithmic loss> similarly gives losses $-\ln(0.3)\approx1.203973$ and $-\ln(0.8)\approx0.223144$. The honest forecast loses this comparison on that draw, which occurs with <probability> $0.3$ in the stipulated <statistical model>.

\Image[/proper-score-realized-outcomes.png]
{title=Individual outcomes can favor a different report even though the expected <Brier score> and <logarithmic loss> are uniquely minimized at $p=0.7$}

The $Y=0$ loss curves favor $p=0$, and the $Y=1$ loss curves favor $p=1$. Their weighted average, with weights $0.3$ and $0.7$, favors $p=0.7$. Choosing a report after observing $Y$ would use information unavailable at forecast time. Even an average loss on a finite dataset can rank an honest forecast below a competing forecast through outcome variation; <strict propriety> guarantees the ranking in <expected value> under the specified information and true <predictive distribution>.