 Entropy functional

For a nonnegative integrable random variable $Z$ on a probability space with probability measure $P$, its entropy functional isusing $0\log 0=0$ and allowing the value $+\infty$. If $m>0$, define the probability measure $Q$ by $dQ/dP=Z/m$. Substitution into the definition of relative entropy givesThus the normalization factor $m$ is essential. Jensen inequality shows that the entropy functional is nonnegative; it vanishes precisely when $Z$ is constant almost surely. If $m=0$, nonnegativity forces $Z=0$ almost surely and the entropy functional is zero.

For an event $A$ of probability $p>0$, take $Z=1_A$. Then $Q=P(\cdot \mid A)$ and ${\mathrm{Ent}}_P(Z)=p\log (1/p)$. This example connects the entropy functional to the information cost of conditioning on a rare event.