Entropy functional
= Entropy functional
{title2=$\operatorname{Ent}(Z)$}
For a nonnegative <integrable random variable> $Z$ on a <probability space> with <probability measure> $P$, its <entropy functional> is
$$
\operatorname{Ent}_P(Z)=\mathbb E_P[Z\log Z]-m\log m,\qquad m=\mathbb E_P Z,
$$
using $0\log0=0$ and allowing the value $+\infty$. If $m>0$, define the <probability measure> $Q$ by $dQ/dP=Z/m$. Substitution into the definition of <relative entropy> gives
$$
\operatorname{Ent}_P(Z)=mD(Q\Vert P).
$$
Thus the normalization factor $m$ is essential. <Jensen inequality> shows that the <entropy functional> is nonnegative; it vanishes precisely when $Z$ is constant <almost surely>. If $m=0$, nonnegativity forces $Z=0$ <almost surely> and the <entropy functional> is zero.
For an <event> $A$ of <probability> $p>0$, take $Z=\mathbf1_A$. Then $Q=P(\cdot\mid A)$ and $\operatorname{Ent}_P(Z)=p\log(1/p)$. This example connects the <entropy functional> to the information cost of conditioning on a rare <event>.