Past exam of the mathematics course of the University of Cambridge / 2006 / iii / Paper 30 / 3

Write $A_2(N,\delta )$ for the largest number of codewords in a binary length-$N$ code with minimum Hamming distance at least $\delta$. The volume of a Hamming ball does not depend on its centre and isFor the Hamming bound, put $r=\lfloor (\delta -1)/2\rfloor$. The radius-$r$ Hamming balls around distinct codewords are disjoint: a point belonging to two such balls would, by the triangle inequality for Hamming distance, put their centres at distance at most $2r<\delta$. Counting points in the binary cube therefore gives

For the Gilbert–Varshamov bound, choose a code maximal under inclusion among those with minimum Hamming distance at least $\delta$, for example by successively adding any allowable word. Its radius-$(\delta -1)$ Hamming balls cover the cube. Otherwise an uncovered word would be at distance at least $\delta$ from every codeword and could be added, contradicting maximality. ThusThe lower bound is an existence assertion about a suitable code, while the upper bound holds for every such code. Together they give

Let $h_2(x)=-x{\log }_{2}x-(1-x){\log }_2(1-x)$ be the binary entropy. For $\delta =\lfloor \lambda N\rfloor$,The assumed Hamming ball volume exponent applies also to any integer radii $d_N$ with $d_N/N\to \theta \in (0,1/2)$. Indeed, for each small $\epsilon >0$ these radii eventually lie between $\lfloor (\theta -\epsilon )N\rfloor$ and $\lfloor (\theta +\epsilon )N\rfloor$. monotonicity of $v_N$ sandwiches its normalized logarithm between limits $h_2(\theta -\epsilon )$ and $h_2(\theta +\epsilon )$; continuity then lets $\epsilon$ tend to zero. Consequently the asymptotic Gilbert–Varshamov bound and asymptotic Hamming bound have respective exponential scalesEquivalently, the optimal code rate satisfiesThese are bounds on the rate, not a claim that the optimal rate has a known limit or equals either endpoint.

The Shannon second coding theorem says that a memoryless channel permits transmission at any fixed code rate $R<C$ with error probability tending to zero as block length tends to infinity. Conversely, a sequence of codes with error probability tending to zero cannot have a limiting rate above $C$. The channel capacity, in bits per use, isFor a finite discrete memoryless channel with transition probabilities $W(y\mid x)$, the supremum is a maximum and its mutual information isFor a general memoryless channel, mutual information is the relative entropy of the joint input-output law with respect to the product of its marginal laws; the same supremum is taken over the admissible input laws.

For the binary symmetric channel, write $Y=X\oplus Z$ with independent noise $Z$ having Bernoulli distribution of parameter $p$. Given either input bit, the output conditional entropy is $h_2(p)$. ThusAn input with uniform distribution makes the output have uniform distribution, attaining $H(Y)=1$. The binary symmetric channel capacity is therefore

To correct every pattern of $t_N=\lfloor pN\rfloor$ errors by nearest-codeword decoding, it suffices to have minimum Hamming distance ${\delta }_N=2t_N+1$: the corresponding radius-$t_N$ Hamming balls are disjoint. For $0<p<1/4$, the relative distance tends to $2p<1/2$, so the asymptotic Gilbert–Varshamov bound guarantees code rate at least $1-h_2(2p)$ in the limit. Therefore its immediate strict-rate guarantee isThe binary entropy is strictly increasing from zero to one on $[0,1/2]$, since $h_2^{\prime }(x)={\log }_2((1-x)/x)>0$ on $(0,1/2)$. Let $a=h_2^{-1}(1-R)$ denote the inverse on this interval. The asymptotic condition becomes

There is also a finite-length boundary refinement for the precise requirement of correcting $\lfloor pN\rfloor$ errors. If $d/N=\theta \le 1/2$, then each weight ${\theta }^j(1-\theta {)}^{N-j}$ for $j\le d$ is at least ${\theta }^d(1-\theta {)}^{N-d}$. Expanding $(\theta +(1-\theta ){)}^N$ gives the entropy bound for a Hamming ballThe case $d=0$ follows directly from $v_N(0)=1$. Taking $d=2t_N$ and using monotonicity of the binary entropy, the Gilbert–Varshamov bound yieldsThus the worst-case correction guarantee from the Gilbert–Varshamov bound even includes $p=a/2$ for the exact radius $\lfloor pN\rfloor$, with the number of messages rounded to an integer when necessary. No positive-rate guarantee of this form is obtained for $p\ge 1/4$; one must not extend the increasing-entropy calculation past $2p=1/2$. Failure of this sufficient bound does not rule out better codes.

By contrast, the Shannon second coding theorem gives vanishing error probability on the binary symmetric channel wheneverThe factor of two comes from demanding disjoint correction balls for every error pattern of a prescribed weight. This is a worst-case distance requirement, and the Gilbert–Varshamov bound is itself only a sufficient existence bound. The Shannon second coding theorem controls the probability of decoding failure for the random channel; rare bad patterns may remain.

Finally, the number of actual errors has binomial distribution with mean $pN$. Correcting exactly its mean does not itself imply error probability tending to zero. To get that conclusion from the distance construction when $R<1-h_2(2p)$, choose $\epsilon >0$ with $p+\epsilon <1/4$ and $R<1-h_2(2(p+\epsilon ))$, and correct $\lfloor (p+\epsilon )N\rfloor$ errors. The weak law of large numbers then makes the probability of exceeding the correction radius tend to zero.