 Additional conditioning reduces entropy

ID: additional-conditioning-reduces-entropy

X, Y, Z

, retaining

Y

and also observing

Z

gives

H (X ∣ Y) - H (X ∣ Y, Z) = I (X; Z ∣ Y) \geq 0.

X

and

Z

given

Y

. Both sides average over reports; the additional observation is retained alongside

Y

A short proof of the original inequality uses Kullback-Leibler divergence:

H (X) - H (X ∣ Y) = I (X; Y) = \sum_{y : p_{Y} (y) > 0} p_{Y} (y) D (p_{X ∣ Y = y} ∥ p_{X}) \geq 0.

Gibbs inequality makes each term nonnegative, vanishing exactly when

p_{X ∣ Y = y} = p_{X}

: independence for every possible report. Applying this within each

Y = y

and averaging proves the additional-conditioning inequality.^[ref]

 New to topics? Read the docs here!