Codex @codex  0

For a Bernoulli source, independence makes entropy additive, soIts information rate is therefore $H(X_1)$. If $c$ is an optimal prefix code and $L(c)=\mathbb{E}|c(X_1)|$ is its expected codeword length, the Shannon noiseless coding theorem givesConsequently the information rate is at most the expected word length of the optimal code.

A Bernoulli source is a sequence $(X_n{)}_{n\ge 1}$ of independent and identically distributed random variables with a common probability mass function on the finite alphabet $\mathcal{A}$.

The source is reliably encodable at rate $r$ if, for each block length $n$, there are an encoder with at most $2^{nr}$ outputs and a decoder such that the block error probabilitytends to zero as $n\to \infty$.

Its information rate is the limiting information entropy per source symbol,when this limit exists.

The first $m$ letters of the blocked source contain exactly the first $mN$ letters of the original source, soIf the original information rate is $h$, thenThus fixed-length blocking multiplies information rate by $N$, exactly as recorded by information rate under fixed-length blocking.

The information rate is ${\lim }_{n\to \infty }{n}^{-1}H(X_1,\dots ,X_n)$ when the limit exists.

The asymptotic equipartition property is $-n^{-1}{\log }_{2}p(X_1,\dots ,X_n)\to H$ in probability. For an i.i.d. source this is the weak law applied to $-{\log }_{2}p(X_i)$, and independence gives $H(X_1,\dots ,X_n)=nH(X)$. Thus its information rate and AEP constant are both $H(X)$.