= Joint entropy
{title2=$H(X_1,\ldots,X_n)$}
<Joint entropy> is the <information entropy> of an entire <random vector>, treating its coordinates together as one outcome. For <random variables> with finite <alphabets> and joint <probability mass function> $p$,
$$
H(X_1,\ldots,X_n)=-\sum_{x_1,\ldots,x_n:p(x_1,\ldots,x_n)>0}p(x_1,\ldots,x_n)\log_2 p(x_1,\ldots,x_n).
$$
It is measured in <bits>. The distinction from separate <information entropies> is whether shared uncertainty is counted once or repeatedly. Two copies of the same fair <bit> have <joint entropy> one <bit>, since only $00$ and $11$ can occur. Two <independent> fair <bits> have <joint entropy> two <bits>, since all four pairs are possible and equiprobable.
For a less symmetric example, assign <probabilities> $1/2,1/4,1/4$ to $(X,Y)=(0,0),(1,0),(1,1)$ respectively. Then
$$
\boxed{H(X,Y)=\tfrac12\cdot1+\tfrac14\cdot2+\tfrac14\cdot2=\tfrac32\text{ bits}.}
$$
Here $X$ is fair, while $\mathbb P(Y=1)=1/4$. Thus $H(X)=1$ and $H(Y)=h_2(1/4)\approx0.811278$ <bits>, where $h_2$ is <binary entropy>. Adding the separate <information entropies> would give approximately $1.811278$ <bits>, overcounting by the <mutual information> $I(X;Y)\approx0.311278$ <bits>.
The <chain rule for information entropy> makes the shared information explicit:
$$
H(X,Y)=H(X)+H(Y\mid X).
$$
In this example $Y$ is certain given $X=0$ and fair given $X=1$, so its average <conditional entropy> is $H(Y\mid X)=\tfrac12\cdot0+\tfrac12\cdot1=1/2$ <bit>. Sending the first coordinate changes how much uncertainty remains in the second. A <prefix code> with codewords `0`, `10`, `11` for the three possible pairs has average length $3/2$ <bits>, exactly the <joint entropy> in this stipulated law.
For finite <alphabets>,
$$
\max_iH(X_i)\leq H(X_1,\ldots,X_n)\leq\sum_iH(X_i).
$$
The lower bound follows because knowing the full <random vector> determines every coordinate; the upper bound follows from <conditioning reduces entropy> in the <chain rule for information entropy>. Equality in the upper bound means joint <independence>. The difference from that upper bound is <total correlation>, whose <noisy parity total correlation> example shows why separate pairwise comparisons need not find all the dependence.
Back to article page
