= Conditional entropy
{title2=$H(Y\mid X)$}
{wiki}
For finite <discrete random variables>, conditional entropy is the average remaining <information entropy> of $Y$ after the value of $X$ is known:
$$
H(Y\mid X)=\sum_{x:p_X(x)>0}p_X(x)H(Y\mid X=x).
$$
Each observation $X=x$ changes the <probability distribution> used to describe $Y$; the average weights that uncertainty by how often the observation occurs. An observation can increase uncertainty for one particular outcome, even though <conditioning reduces entropy> on average for finite variables.
For example, let $Y$ choose $A,B,C,D$ with <probabilities> $1/2,1/4,1/8,1/8$, and let $X$ report whether $Y=A$. If it reports yes, there is no remaining uncertainty. Otherwise the conditional <probabilities> are $1/2,1/4,1/4$, whose <information entropy> is $3/2$ <bits>. The two reports each occur with <probability> $1/2$, so $H(Y\mid X)=3/4$ of a <bit>. The original uncertainty was $7/4$ <bits>: this observation removes exactly one <bit> on average.
If $Y$ is a <function> of $X$, then $H(Y\mid X)=0$: the observation determines the answer. If $X$ and $Y$ are <independent random variables>, then $H(Y\mid X)=H(Y)$: the observation gives no improvement. These endpoints make <conditional entropy> useful for separating relevant observations from irrelevant ones. They also lead to the <chain rule for information entropy>, $H(X,Y)=H(X)+H(Y\mid X)$, which counts the first uncertainty and the remaining uncertainty without double counting.
Back to article page
