Logarithmic scoring rule (source code)

= Logarithmic scoring rule
{title2=$S(p,y)=\log p(y)$}

The <logarithmic scoring rule> rewards the <natural logarithm> of the reported <probability mass> or <probability density> at the observed outcome. The title uses the reward convention. For a binary report $p$, its negative is the <logarithmic loss>:
$$
L_{\log}(p,y)=-y\ln p-(1-y)\ln(1-p),\qquad 0<p<1.
$$
For $y=1$ this is $-\ln p$; for $y=0$ it is $-\ln(1-p)$. At an endpoint, use these outcome-specific forms: assigning zero <probability> to the outcome incurs infinite loss, while assigning it <probability> one gives zero. The <natural logarithm> therefore makes a confidently wrong binary forecast far more costly than the bounded <Brier score> does.

Under the stipulated <Bernoulli distribution> with $q=0.7$, the expected logarithmic loss is
$$
\mathbb E_{0.7}[L_{\log}(p,Y)]=-0.7\ln p-0.3\ln(1-p).
$$
Its unique minimum is at $p=0.7$, where the expected loss is approximately $0.610864$. At $p=0.2$ it is approximately $1.193550$. More generally, the excess expected logarithmic loss over truthful reporting is the <Kullback-Leibler divergence> in the order true law first, reported law second:
$$
\mathbb E_q[L_{\log}(p,Y)]-\mathbb E_q[L_{\log}(q,Y)]
=q\ln\frac qp+(1-q)\ln\frac{1-q}{1-p}
=D_{\rm KL}\!\left(\operatorname{Bernoulli}(q)\Vert\operatorname{Bernoulli}(p)\right).
$$
For $0<q<1$, this <Kullback-Leibler divergence> is nonnegative and vanishes only at $p=q$, explaining why the <logarithmic scoring rule> is a <strictly proper scoring rule>. The same <Kullback-Leibler divergence> identity applies to <probability density functions> relative to a common <measure> when the truthful expected <logarithmic loss> is finite and the comparison <expected value> is well defined.<Strictly Proper Scoring Rules, Prediction, and Estimation by Tilmann Gneiting and others>[Section 3.1, Example 3, and Section 4.1]{ref} For sequential forecasts, the existing <prequential log score identity> explains how logarithmic rewards add over observations.