 Relative entropy and coordinate dependence

Apply the same invertible differentiable transformation to two probability distributions with densities $f$ and $r$. Both transformed densities acquire the same Jacobian determinant factor. Write their transformed densities as $f_g$ and $r_g$. Their ratio is unchanged, and the integration measure transforms with the numerator density. Consequently their relative entropy is unchanged:The densities describe two alternative laws in the same coordinate, with a nonzero Jacobian determinant on their common domain, and the expression is assumed well defined. This explains why comparing two laws can survive a change of measurement units even though either law's differential entropy changes. Mutual information, formulated as relative entropy from a product of marginals, has the same invariance under invertible changes to either observed coordinate.

Differential entropy is also the entropy term in the evidence lower bound for continuous variational inference. Its coordinate dependence must be combined consistently with that of the model density; changing units does not justify changing only one term of the objective.