= Information from two conditionally independent observations
{title2=$I(U;V,W)=I(U;V)+I(U;W)-I(V;W)$}
If outputs $V,W$ are <conditionally independent> given $U$, then $H(V,W\mid U)=H(V\mid U)+H(W\mid U)$. Expanding the <Shannon entropy>[entropy] expressions proves the displayed <mutual information> identity. The chain rule also gives $I(U;V,W)\geq\max\{I(U;V),I(U;W)\}$, while the identity and nonnegativity give the upper bound by their <sum>. For two <Communication channel>[channels] with common input and <independent> output noise conditional on that input, maximizing over the input law therefore bounds the joint-output <Channel capacity>[capacity] between the larger marginal <Channel capacity>[capacity] and the <sum> of the two marginal capacities. Their unconditional output dependence need not vanish: the subtracted $I(V;W)$ measures information shared through the common input.
Back to article page
