 Relative entropy nonnegativity

For probability distributions $P,Q$, relative entropy satisfies $D_{\mathrm{KL}}(P\Vert Q)\ge 0$, with equality exactly when $P=Q$. Thus truthful prediction uniquely minimizes expected log-loss whenever the comparison is well defined, explaining the strictly proper scoring rule property of the logarithmic scoring rule.

For a short proof with probability density functions $f,g$ and $g>0$, set $r=f/g$. Their integrals both equal one, givingThe strictly convex function $r\log r-r+1$ is minimized only at $r=1$, so equality means $f=g$ almost everywhere. Its value at zero is one by $0\log 0=0$. Where $g=0<f$, relative entropy is infinite; where both vanish, the contribution is zero. These conventions extend the argument beyond positive reference probability density functions.