= Correlated binary variables
{title2=$B=A\oplus N,\quad H(A,B)=1+h_2(p)$}
Let $A$ be a fair <bit>, let $N$ be an <independent> <Bernoulli random variable> with $\mathbb P(N=1)=p$, and set $B=A\oplus N$, where $\oplus$ is <exclusive or>. Thus $N=1$ flips the first <bit>. This constructs a <binary symmetric channel> with a fair input, for $0\leq p\leq1$. The <joint probability mass function> is
$$
\begin{array}{c|cc|c}
& B=0&B=1&\mathbb P(A=a)\\\hline
A=0&(1-p)/2&p/2&1/2\\
A=1&p/2&(1-p)/2&1/2\\\hline
\mathbb P(B=b)&1/2&1/2&1
\end{array}
$$
The column sums show that $B$ is also a fair <bit>, whatever $p$ is. Both <marginal distributions> therefore have <information entropy> one <bit>. Given $A$, only the flip decision remains unknown, giving <conditional entropy> $H(B\mid A)=h_2(p)$, where
$$
h_2(p)=-p\log_2p-(1-p)\log_2(1-p)
$$
is the <binary entropy>, with $h_2(0)=h_2(1)=0$. The <chain rule for information entropy> and the definition of <mutual information> now give
$$
\boxed{H(A,B)=1+h_2(p),\qquad I(A;B)=1-h_2(p).}
$$
Take $p=1/10$. The four joint <probabilities>, in row order, are $(0.45,0.05,0.05,0.45)$. Directly from the definition of <joint entropy>,
$$
H(A,B)=-2(0.45)\log_2(0.45)-2(0.05)\log_2(0.05)
\approx1.468995594\ \text{bits}.
$$
Conditioning on either value of $A$ gives <probabilities> $0.9$ and $0.1$ for the two values of $B$, hence
$$
H(B\mid A)=h_2(0.1)\approx0.468995594\ \text{bits},\qquad
I(A;B)\approx0.531004406\ \text{bits}.
$$
Adding $H(A)+H(B)=2$ would overcount by exactly $I(A;B)$. These are average uncertainties under the stipulated <probability distribution>; the four possible pairs are still represented by four distinct outcomes.
\Image[/entropy-chain-rule-joint-table.png]
{title=A correlated binary pair contains one bit plus the remaining flip uncertainty}
{description=Original exact-model <joint probability mass function> for $p=0.1$. The bars compare the sum of marginal <information entropies> with the <joint entropy> from the <chain rule for information entropy>. The hatched portion is the <mutual information> counted twice by the marginal sum. Values are calculated, not empirical frequencies.}
There is another way to see the decomposition: $(A,B)\mapsto(A,A\oplus B)=(A,N)$ is a <bijection>. It merely relabels the four outcomes, so it preserves <joint entropy>. The <independence> of $A$ and $N$ then gives $H(A,B)=H(A,N)=H(A)+H(N)=1+h_2(p)$. Describing the initial <bit> and the flip decision separates shared uncertainty from new uncertainty.
At $p=0$, the <bits> are identical: $B=A$, so $H(B\mid A)=0$ and $I(A;B)=1$. At $p=1$, they are complementary: $B=1-A$, with exactly the same <conditional entropy> and <mutual information>. In either case, observing one determines the other. At $p=1/2$, all four joint <probabilities> are $1/4$; the <bits> are <independent>, $H(A,B)=2$, and $I(A;B)=0$. Perfect disagreement is therefore just as informative as perfect agreement.
\Image[/entropy-chain-rule-dependence-curves.png]
{title=Identical and complementary bits both reveal one full bit of information}
{description=Original calculated curves for the constructed pair $B=A\oplus N$: <joint entropy> $1+h_2(p)$, <conditional entropy> $h_2(p)$, and <mutual information> $1-h_2(p)$, all in <bits>. The horizontal line marks the one-bit <information entropy> of either <marginal distribution>. Reflection about $p=1/2$ exchanges likely agreement with likely disagreement while preserving these quantities.}
For $p>1/2$, guessing $A=1-B$ succeeds with <probability> $p$, better than guessing $A=B$. Changing $B$ to $1-B$ replaces $p$ by $1-p$ and preserves <mutual information>, since the relabeling is invertible. What matters is how well the observation distinguishes the input, rather than how often its symbol matches the input. The same symmetry makes the <binary entropy> curve peak at <independence>, while the <mutual information> curve reaches its minimum there.
Back to article page
