= Relative entropy nonnegativity
{title2=$H(f\mid g)\geq0$}
For <probability distributions> $P,Q$, <relative entropy> satisfies $D_{\mathrm{KL}}(P\Vert Q)\geq0$, with equality exactly when $P=Q$. Thus truthful prediction uniquely minimizes expected <log-loss> whenever the comparison is well defined, explaining the <strictly proper scoring rule> property of the <logarithmic scoring rule>.
For a short proof with <probability density functions> $f,g$ and $g>0$, set $r=f/g$. Their <integrals> both equal one, giving
$$
H(f\mid g)=\int g\bigl(r\log r-r+1\bigr)\,d\mu\geq0.
$$
The <strictly convex function> $r\log r-r+1$ is minimized only at $r=1$, so equality means $f=g$ <almost everywhere>. Its value at zero is one by $0\log0=0$. Where $g=0<f$, <relative entropy> is infinite; where both vanish, the contribution is zero. These conventions extend the argument beyond positive reference <probability density functions>.
Back to article page
