= Stationarity without ergodicity in an information source
Choose one of two regimes once, with equal <probability>: a source of <independent and identically distributed random variables> with one-bit probability $1/10$, or one with one-bit probability $1/2$. Keep the chosen regime forever. The resulting source is a <strictly stationary process>, because shifting a block does not change either component law. It is not an <ergodic stationary process>: its limiting empirical frequency identifies the regime.
The source's <information rate> is
$$
\frac12h_2(1/10)+\frac12h_2(1/2)\approx0.734498.
$$
Conditioning on the single regime bit changes any block's <information entropy> by at most one <bit>, so dividing by the block length proves this rate. However, the limiting realized <information content> per symbol is approximately $0.468996$ in the first regime and one in the second. The alternative component's <likelihood ratio> decays exponentially along a typical path from the chosen component, at a rate given by the positive <relative entropy> between the two <Bernoulli distributions>. The mixture therefore has two possible pathwise limits, not one limit equal to the averaged <information rate>.
Back to article page
