= Equality in the entropy conditioning inequality
For <discrete random variables> with finite <information entropy> $H(X)$, <Gibbs inequality> gives
$$
H(X)-H(X\mid Y)=\sum_y p_Y(y)D(p_{X\mid y}\Vert p_X)\geq0.
$$
Equality holds exactly when $X$ and $Y$ are <independent random variables>. Finiteness matters: if $A$ has infinite <information entropy> and is independent of a fair bit $B$, then $X=(A,B)$ and $Y=B$ are dependent but $H(X)=H(X\mid Y)=+\infty$.
// Target: probability-and-statistics.bigb
Back to article page
