= Conditioning and differential entropy
With appropriate <probability density functions> and finite <differential entropies>, $h(X)-h(X\mid Y)=I(X;Y)\geq0$, with equality iff <independence>. <Differential entropy> does not reverse the inequality; infinite or undefined quantities cannot justify subtraction.<6.441 Information Theory (Spring 2016 lecture notes) by Yury Polyanskiy and others/Chapter 2: Information Measures: Mutual Information>[Theorem 2.4(3), printed p. 24: density and finiteness hypotheses for the differential-entropy identity]{ref}
However, <differential entropy> can be negative and depends on the choice of coordinates. For $X$ with a <uniform distribution> on an interval of length $a>0$, $h(X)=\log_2a$; in particular $a=1/2$ gives $-1$. Rescaling $X$ by a constant $c>0$ adds $\log_2c$ to both $h(X)$ and $h(X\mid Y)$, whenever they are finite, leaving the <mutual information> unchanged.<A Mathematical Theory of Communication by Claude Shannon>[pp. 37–38: coordinate dependence, negative entropy, and linear rescaling]{ref}
If a continuously distributed $X$ is revealed exactly, its <conditional distribution> is a <Dirac measure> with no ordinary <probability density function>. The discrete identity $H(X\mid X)=0$ has no finite <differential entropy> counterpart. <Mutual information>, defined through <relative entropy>, is infinite here.<6.441 Information Theory (Spring 2016 lecture notes) by Yury Polyanskiy and others/Chapter 2: Information Measures: Mutual Information>[Theorem 2.4(1) and its proof, printed p. 24: I(X;X) is infinite for a non-discrete distribution]{ref}
Back to article page
