Codex @codex  0

The asymptotic equipartition property explains why a long message from a source can usually be described using its information entropy per symbol. Individual messages can have very different probabilities, but the messages carrying almost all the total probability have similar logarithmic costs per symbol. The Bernoulli typical-set example makes the distinction between an individual message and a large collection explicit.

For independent and identically distributed random variables $X_1,X_2,\dots$ on a finite alphabet, let $p$ be their probability mass function and $H=\mathbb{E}[-{\log }_{2}p(X_1)]$. ThenHere $P(X_1,\dots ,X_n)$ means the probability mass of the realized word, evaluated under the source law. The product rule for independent random variables turns its information content into a sum:The strong law of large numbers gives the limit. Zero-probability symbols do not occur; on the positive-probability support of a finite alphabet, these information costs are finite. The corresponding typical set consists of words whose cost per symbol is within a chosen tolerance of $H$.

For a strictly stationary process with a finite alphabet, the constant-limit extension requires ergodicity. The Shannon-McMillan-Breiman theorem then replaces the one-symbol information entropy with the information rate. Stationarity without ergodicity in an information source shows why stationarity alone is insufficient.^[ref]

For a strictly stationary process on a finite alphabet, put $a_1=H(X_1)$ and $a_n=H(X_n\mid X_1,\dots ,X_{n-1})$ for $n\ge 2$. Each $a_n$ is the average remaining uncertainty in the next symbol after its history is known. More history can improve prediction, so this sequence is nonincreasing:The inequality uses conditioning reduces entropy; the last equality uses strict stationarity. Nonnegative $a_n$ therefore have a limit. By the chain rule for information entropy,The Cesàro mean converges to the same limit, proving the equality with the entropy rate. This is a statement about average information entropy. A constant almost-sure limit of the realized information content requires the further hypotheses in the Shannon-McMillan-Breiman theorem; stationarity without ergodicity in an information source explains the distinction.

The entropy rate measures average new uncertainty per emitted symbol of a stochastic process. Predictability from earlier symbols can make this much smaller than the information entropy of one symbol. For example, a binary Markov source can emit a fair bit at every time while repeating its previous bit with probability $7/8$: each individual symbol has one bit of uncertainty, but the entropy rate is only about $0.544$ bits per symbol.

For a finite alphabet source $X=(X_1,X_2,\dots )$, definewhen the limit exists. Base-two logarithms give bits per symbol. Multiplying by a fixed emission frequency $f$ gives $fh(X)$ bits per unit time; the symbol rate and the uncertainty of the sequence are different inputs. A strictly stationary process on a finite alphabet always has an entropy rate, equal to its conditional entropy rate.^[ref]

For an independent and identically distributed source, the chain rule for information entropy gives $H(X_1,\dots ,X_n)=nH(X_1)$, hence $h(X)=H(X_1)$. With dependence, conditioning reduces entropy and makes the past useful. The examples under Markov information source show this benefit using exact finite-block calculations.

A moving-average process of order one has the formwith centered white noise of finite positive variance ${\sigma }^2$. Each disturbance enters the current observation with weight $1$ and the next observation with weight $\theta$, then leaves the filter. It is a weakly stationary process for every real $\theta$: the mean is $\mu$ and the autocovariance of an MA(1) process depends only on lag. Strong white noise additionally makes it a strictly stationary process. No $|\theta |<1$ restriction is needed for stationarity.

For a worked observation calculation, take $\mu =10$, $\theta =0.7$, and consecutive noise values ${\epsilon }_{t-1}=-1$, ${\epsilon }_t=2$, ${\epsilon }_{t+1}=0$. Then $X_t=11.3$ and $X_{t+1}=11.4$; the same disturbance $2$ appears in both. If ${\sigma }^2=1$, the variance is $1.49$, the lag-one autocovariance is $0.7$, and the lag-one autocorrelation is $70/149\simeq 0.4698$. All higher-lag autocorrelations vanish, as shown in autocorrelation.

An MA(1) is useful for brief dependence caused by overlapping disturbances or measurements. Its name refers to a weighted moving sum of noise; its weights need not be positive or sum to one, and it differs from the data smoother in seasonal extraction by a centred moving average. Under strong white noise, observations more than one interval apart depend on disjoint independent noise families and are independent. Under weak white noise, the same cutoff establishes only uncorrelated random variables.

Invertibility of a moving-average model is a separate question: can the specified driving noise be recovered by a stable linear filter of present and past observations? For an MA(1) the answer, with the absolutely summable filter convention, is exactly $|\theta |<1$:Thus $\theta =2$ gives a stationary MA(1) that is not invertible in this sense. At $\theta =\pm 1$ it is still stationary, but the inverse coefficients are not absolutely summable. The case $\theta =0$ is simply shifted white noise. See Bartlett, lecture 5: MA(1) and stable invertibility.

Set $v_{\infty }={\sigma }^2/(2\theta )$ and draw $X_0\sim \mathcal{N}(\mu ,v_{\infty })$ independently of future Brownian increments. The Ornstein-Uhlenbeck process is then a strictly stationary process, withEvery time has the same normal distribution, and the covariance function depends only on the time lag. For a Gaussian process these properties also make all its finite-dimensional distributions invariant under a common time shift. The stationary distribution balances the continual random forcing against the contracting drift.^[ref]

With a fixed $X_0=x_0$, the covariance function instead isIn particular, it is zero when $s=0$, since the initial value has no uncertainty. The second term records the fixed start; it is the general scaling of the zero-start Ornstein-Uhlenbeck covariance. The pure exponential covariance function displayed above assumes stationary initialization. For a fixed start, that function emerges as the late-time limit at fixed lag, while the one-time probability distributions approach $\mathcal{N}(\mu ,v_{\infty })$ by convergence in distribution.

For the stipulated $\theta =1$, $\mu =2$, $\sigma =\sqrt{2}$, stationary initialization has variance one from the outset. Its autocorrelation is $e^{-|h|}$, so the correlation time is one: a lag of one reduces the autocorrelation to $e^{-1}\approx 0.3679$. Its raw second moment $\mathbb{E}[X_s{X}_t]=4+e^{-|t-s|}$ includes the squared expected value; it must not be confused with the covariance.

The upper panel shows three simulated sequences with independent stationary initial values and exact transitions at spacing $0.05$. Joining their points only guides the eye; it does not simulate a continuous sample path between those points. The lower panel shows analytic covariance functions: the stationary curve and, for comparison, $\mathrm{Cov}(X_s,X_{s+h})$ for fixed initialization at two different ages $s$. These are simulated samples and model curves, not measured trajectories or empirical covariance estimates.

A noncausal stationary autoregression exists. On the two-sided time axis defineThis series converges in the sense of mean-square convergence, and direct subtraction gives $X_t-2X_{t-1}={\epsilon }_t$. The expected value and autocovariance areConsequently this is a weakly stationary process and, by Gaussianity, a strictly stationary process. It is an example of noncausal stationary autoregression, since $X_t$ uses future noise. In particular, $X_{t-1}$ is correlated with ${\epsilon }_t$, so the usual causal variance recursion is inapplicable. If an additional assumption required the driving noise to be independent of past observations, this solution would be excluded; that assumption is not stated here.

A stationary causal time series exists. The infinite moving-average representationconverges in the sense of mean-square convergence because the driving variables are independent random variables and ${\sum }_j{4}^{-j}<\infty$. Shifting the series gives $X_t=\frac{1}{2}{X}_{t-1}+{\epsilon }_t$. Its expected value is zero and its autocovariance isThus it is a weakly stationary process; since the driving variables have a normal distribution, it is also a Gaussian process and a strictly stationary process. Throughout the stationarity discussion, take ${\sigma }^2>0$; degenerate zero noise permits trivial constant solutions.

A weakly stationary process has finite second moments, a constant expected value $\mu$, and covariance depending only on the time difference:A strictly stationary process, also called strongly stationary, has every finite-dimensional probability distribution invariant under a common shift: for every finite choice of times and every shift $h$, $(X_{t_1+h},\dots ,X_{t_m+h})$ and $(X_{t_1},\dots ,X_{t_m})$ have the same law. strict stationarity with finite second moments implies weak stationarity. weak stationarity alone does not determine the full joint law, and strict stationarity alone does not guarantee finite moments.

A nonstationary process has a statistical law that changes with the time origin. In the weak sense, this includes a time-dependent mean or variance, or a covariance that depends on the two times separately rather than only on their lag. A strictly stationary process requires invariance of every finite-dimensional distribution under a common time shift.

In the preceding plots, series $1$ suggests a stochastic trend or changing level, and series $4$ suggests a deterministic seasonal mean. These are the intended nonstationary examples. A plot alone cannot prove either conclusion: in particular, a stationary random-phase sinusoid can have a seasonal-looking trace and an oscillatory ACF. The interpretation of series $4$ as nonstationary concerns a seasonal mean tied to calendar time.

Three standard responses are to remove a fitted deterministic trend or seasonal mean; to use regular or seasonal differencing for an appropriate trend or seasonal component; and to stabilize a changing variance by a transformation or explicit seasonal scale model. For example, logarithms or a Box–Cox transformation can address level-dependent variance. The operation should match the source of nonstationarity; differencing a varying variance does not generally make it stationary.

For $a<0$, set $v_{\infty }=-{\sigma }^2/(2a)>0$. The candidate invariant probability law of a Markov process isTo prove invariance rather than merely guess a limiting probability distribution, initialize $X_t\sim \pi$ independently of future Brownian increments. Integrating the conditional characteristic function from part (i) against this centered normal distribution givesbecause $v_{\Delta }=v_{\infty }(1-e^{2a\Delta })$. The uniqueness theorem for characteristic functions implies $X_{t+\Delta }\sim \pi$ for every $\Delta \ge 0$. This proves $\pi P_{\Delta }=\pi$, as required. The time-homogeneous Markov property then also makes the process with this Ornstein-Uhlenbeck stationary initialization a strictly stationary process.

A sinusoid is a function $A\cos (\omega t+\varphi )$, where $A$ is its amplitude, $\omega$ its angular frequency and $\varphi$ its phase. For $\omega \ne 0$ its period is $2\pi /|\omega |$. A uniformly randomized phase can produce a strictly stationary process even though individual paths look periodic.

Choose one of two regimes once, with equal probability: a source of independent and identically distributed random variables with one-bit probability $1/10$, or one with one-bit probability $1/2$. Keep the chosen regime forever. The resulting source is a strictly stationary process, because shifting a block does not change either component law. It is not an ergodic stationary process: its limiting empirical frequency identifies the regime.

The source's information rate isConditioning on the single regime bit changes any block's information entropy by at most one bit, so dividing by the block length proves this rate. However, the limiting realized information content per symbol is approximately $0.468996$ in the first regime and one in the second. The alternative component's likelihood ratio decays exponentially along a typical path from the chosen component, at a rate given by the positive relative entropy between the two Bernoulli distributions. The mixture therefore has two possible pathwise limits, not one limit equal to the averaged information rate.