= Relative entropy and coordinate dependence
Apply the same invertible differentiable transformation to two <probability distributions> with densities $f$ and $r$. Both transformed densities acquire the same <Jacobian determinant> factor. Write their transformed densities as $f_g$ and $r_g$. Their ratio is unchanged, and the integration measure transforms with the numerator density. Consequently their <relative entropy> is unchanged:
$$
D(f_g\Vert r_g)=D(f\Vert r).
$$
The densities describe two alternative laws in the same coordinate, with a nonzero <Jacobian determinant> on their common domain, and the expression is assumed well defined. This explains why comparing two laws can survive a change of measurement units even though either law's <differential entropy> changes. <Mutual information>, formulated as <relative entropy> from a product of marginals, has the same invariance under invertible changes to either observed coordinate.
<Differential entropy> is also the entropy term in the <evidence lower bound> for continuous <variational inference>. Its coordinate dependence must be combined consistently with that of the model density; changing units does not justify changing only one term of the objective.
Back to article page
