= An observation can increase entropy
Consider a binary state $X$ and a report $Y$ with the following <joint probability mass function>:
$$
\begin{array}{c|cc|c}
&Y=0&Y=1&p_X(x)\\\hline
X=0&4/5&1/10&9/10\\
X=1&0&1/10&1/10\\\hline
p_Y(y)&4/5&1/5&1
\end{array}
$$
Before the report, $X=0$ is strongly favored: its <information entropy> is $H(X)=h_2(1/10)\approx0.468996$ <bits>, where $h_2$ is <binary entropy>. A report $Y=1$ makes the two states equally likely, so $H(X\mid Y=1)=1$ <bit>, strictly larger than before. This report occurs with <probability> $1/5$. The more common report $Y=0$ determines $X=0$, giving $H(X\mid Y=0)=0$. Therefore the averaged <conditional entropy> is
$$
H(X\mid Y)=\frac45\cdot0+\frac15\cdot1=\frac15<h_2(1/10)=H(X).
$$
The resulting <mutual information> is $I(X;Y)=h_2(1/10)-1/5\approx0.268996$ <bits>. The rare report increases uncertainty about the state; that increase does not make the report useless. Together the reports remove uncertainty on average. These are stipulated <probabilities> in an exact model, not measured frequencies.
\Image[/conditioning-entropy-outcome-counterexample.png]
{title=A rare report raises entropy although the weighted average falls}
Back to article page
