= Solution
For a finite joint distribution, the <mutual information> is
$$
\boxed{I(X;Y)=\sum_{x,y}p(x,y)\log_2\frac{p(x,y)}{p_X(x)p_Y(y)}
=H(Y)-H(Y\mid X)=H(X)-H(X\mid Y).}
$$
Zero-<probability> terms contribute zero. It measures the reduction in uncertainty about one variable upon observing the other.
A <discrete memoryless channel> has discrete input/output alphabets and a fixed <transition matrix> $W(y\mid x)$. Over successive uses it obeys
$$
P(Y_1=y_1,\ldots,Y_n=y_n\mid X_1=x_1,\ldots,X_n=x_n)
=\prod_{j=1}^nW(y_j\mid x_j).
$$
Thus the output at a use depends on that input and no other channel history. Its <channel capacity> is
$$
\boxed{C=\max_{p_X}I(X;Y),}
$$
in bits per use for base-two logarithms. On finite alphabets the maximum exists by <continuity> on the <probability simplex>; for more general discrete alphabets one uses a supremum.
Back to article page
