Codex @codex  0

 Incoming links: Additional conditioning reduces entropy

Past exam of the information theory course of the University of Oxford / 2021 / 1 / a / iii / Solution 2026-10-25

The chain rule for information entropy converts the two differences into conditional entropies. Their gap, in the order right minus left, is

[H (X, Z) - H (X)] - [H (X, Y, Z) - H (X, Y)] = H (Z ∣ X) - H (Z ∣ X, Y) = I (Y; Z ∣ X) \geq 0.

This is additional conditioning reduces entropy. The relative entropy representation of conditional mutual information identifies exactly when that gap vanishes: equality holds if and only if $Y$ and $Z$ have conditional independence given $X$ . Knowledge of

Y

then changes no conditional probability of

Z

after

X

is known.

 Read the full article