= Independent revelation of a binary variable
{title2=$H(X\mid Y)=(1-r)h_2(p),\quad I(X;Y)=rh_2(p)$}
Let $X$ be a <Bernoulli random variable> with $\Pr(X=1)=p$. Independently of $X$, a <Bernoulli random variable> $R$ with $\Pr(R=1)=r$ decides whether to reveal it:
$$
Y=\begin{cases}X,&R=1,\\\bot,&R=0.\end{cases}
$$
The symbol $\bot$ means “hidden” and is distinct from both binary states. This is a <binary erasure channel> with erasure <probability> $1-r$. The hidden report leaves the <posterior distribution> equal to the <prior distribution>, because hiding is independent of the state. Either visible report determines $X$. Weighting these three reports gives the exact <conditional entropy> and <mutual information>:
$$
H(X)=h_2(p),\qquad H(X\mid Y)=(1-r)h_2(p),\qquad I(X;Y)=rh_2(p).
$$
For $p=1/4$ and $r=1/2$, $H(X)=h_2(1/4)=2-(3/4)\log_2 3\approx0.811278$ <bits>, while $H(X\mid Y)=I(X;Y)\approx0.405639$ <bits>. With $r=0$ the report tells us nothing; with $r=1$ it reveals the state completely. If $0<p<1$, every positive reveal <probability> makes the conditioning inequality strict. At $p=0$ or $p=1$, the state was already known and every value of $r$ gives equality.
\Image[/entropy-independent-binary-revelation.png]
{title=Independent revelation divides binary entropy into remaining uncertainty and revealed information}
For a <Bayesian inference> problem, this model distinguishes an uninformative missing report from a decisive observed state. If hiding depended on $X$, even $Y=\bot$ could carry <mutual information>, and the displayed calculation would change. In <lossless source coding>, suppose both <encoder> and <decoder> already know the reveal pattern for repeated <independent and identically distributed> pairs. Only the hidden states require descriptions. Applying <Shannon's source coding theorem> to those states gives an asymptotic average rate $(1-r)h_2(p)$ <bits> per original state. A single hidden state need not have a <codeword> of that fractional length.
The reveal pattern itself has <information entropy> $H(R)=h_2(r)$. If that pattern must also be sent, the <chain rule for information entropy> gives $H(R,X)=h_2(r)+h_2(p)$, since $R$ and $X$ are <independent random variables>. Equivalently, $H(Y)=h_2(r)+rh_2(p)$ and $H(X,Y)=H(Y)+H(X\mid Y)=h_2(r)+h_2(p)$. Thus a saving in the description of $X$ given $Y$ assumes that $Y$ is already available; it is not a free saving in the description of the whole pair.
Back to article page
