Differential entropy under an increasing transformation ($h_b(g(X))=h_b(X)+\mathbb{E}{\log }_b{g}^{\prime }(X)$)

Let $X$ have a probability density function on an open interval, and let $g$ be continuously differentiable, strictly increasing there, with $g^{\prime }(x)>0$. The change of variables formula givesIf the differential entropy and the expected absolute logarithm of $g^{\prime }(X)$ are finite, taking the expectation of the negative logarithm proves the displayed formula, for a fixed base $b>1$. Local stretching reduces probability density; its contribution to differential entropy is the expected logarithmic stretch. The map is invertible, so this change does not mean that the transformation destroyed information about $X$.

Take $X$ with continuous uniform distribution on $(0,1)$ and set $Y=X^2$. This is an increasing invertible transformation on that interval. Its cumulative distribution function is $F_Y(y)=\sqrt{y}$, givingAlthough the probability density diverges near zero, it is integrable. The logarithmic stretch is integrable too:The inverse function $X=\sqrt{Y}$ recovers every input in the interval. The changed differential entropy records a changed coordinate volume, rather than a failure of recovery.

// Target: probability-and-statistics.bigb