Asymptotic equipartition property Created 2026-09-24 Updated 2026-10-14
The asymptotic equipartition property explains why a long message from a source can usually be described using its information entropy per symbol. Individual messages can have very different probabilities, but the messages carrying almost all the total probability have similar logarithmic costs per symbol. The Bernoulli typical-set example makes the distinction between an individual message and a large collection explicit.
For independent and identically distributed random variables on a finite alphabet, let be their probability mass function and . Then
Here means the probability mass of the realized word, evaluated under the source law. The product rule for independent random variables turns its information content into a sum:
The strong law of large numbers gives the limit. Zero-probability symbols do not occur; on the positive-probability support of a finite alphabet, these information costs are finite. The corresponding typical set consists of words whose cost per symbol is within a chosen tolerance of .
For a strictly stationary process with a finite alphabet, the constant-limit extension requires ergodicity. The Shannon-McMillan-Breiman theorem then replaces the one-symbol information entropy with the information rate. Stationarity without ergodicity in an information source shows why stationarity alone is insufficient.[ref]
Apply the Birkhoff ergodic theorem to a countable generating algebra, and disintegrate the resulting common full-measure convergence set over the invariant sigma-algebra. The orbit limits on a fiber are its conditional masses. Applying the theorem to that invariant component shows its invariant projection is constant on all generating indicators, hence on all functions by density. This proves ergodicity without an uncountable intersection of null sets.
Ergodic hypothesis 2026-10-13
The ergodic hypothesis identifies long-time averages of equilibrium observables along typical trajectories with averages over the invariant equilibrium ensemble at the same conserved quantities. In the microcanonical formulation, the ensemble is normalized phase-space measure on the energy surface. In a measure-preserving description, ergodicity means invariant measurable sets have measure zero or one; the Birkhoff ergodic theorem then equates time averages and ensemble expectations almost everywhere for integrable observables. This explains why statistical-mechanical ensembles can predict long-time measurements. A particular mechanical model still requires its own ergodicity assumptions; integrable idealized systems need not be ergodic.
Little's law 2026-10-14
For a stable queue with finite long-run averages, a positive finite admitted rate, and negligible initial and terminal contributions after division by observation time, Little's law states
Here is the time-average queue population, is the admitted rate through the chosen system, and is the customer-average customer sojourn time inside that same boundary. Without abandonment, admitted rate equals completed-service queue throughput in this stable regime. It may differ from the offered arrival rate because of blocking. When customers abandon, count their admissions and exits too, and average their actual time inside; using only service completions would mismatch the population being measured.
For the waiting area alone the matching identity is
where is mean queueing delay. For one server with no abandonment, service in arrival order, and finite mean service time ,
with the server utilization. Mixing total population with waiting-only duration would count different portions of each customer's visit. Customers who start service immediately contribute zero to , and must still be included in that customer average.
The strength of Little's law is that it does not require a Poisson process, an exponential distribution of service times, or service in arrival order. The sample-path area identity for Little's law explains why: counting customer-time across time slices or across customer visits gives the same area. A sufficient probabilistic setting is repeated independent and identically distributed empty-to-empty cycles with finite positive mean cycle duration, finite positive mean customer count, and finite mean population area; the renewal-reward theorem then supplies the long-run averages. Stability alone does not guarantee finite mean queueing delay: the Pollaczek-Khinchine mean formula shows why a finite second moment of service time matters for that mean in an M/G/1 queue.
For example, suppose a service desk receives six requests per hour but admits only four; measured mean customer sojourn time of admitted requests is half an hour. Its mean queue population is customers. Multiplying the offered rate by that duration would incorrectly give three. Conversely, counting two customers on average and measuring admitted queue throughput of four per hour gives hour. This lets one estimate a hard-to-track duration from occupancy and completed-job counts.
John D. C. Little published a rigorous proof in 1961 under finite-mean, strict stationarity, and ergodicity assumptions on the arrival process. Those were assumptions of that proof, rather than a requirement that every useful form of the identity have random arrivals. [ref]. [ref].
Identify the circle group with modulo its endpoints, and let be normalized Lebesgue measure. We first check invariance. On the th inverse branch, for . For an integrable function , substitution gives
In particular, testing indicator functions shows that the integer multiplication map on the circle preserves .
To prove ergodicity, let be a measurable set with and put . Then almost everywhere. Its Fourier coefficients are
Using invariance of and substituting separately on the same branches gives
In the last line, the finite geometric series of th roots of unity is zero unless divides , when it equals . For any nonzero integer , divide repeatedly by until the resulting integer is no longer divisible by . The recurrence then implies .
Since the Fourier basis is a complete orthonormal basis of , the absence of every nonconstant Fourier coefficient implies almost everywhere. An indicator function takes only the values zero and one, so must be zero or one. This proves
For the compact set chosen in the preceding step, the Urysohn lemma gives a continuous function equal to one on and zero on . In a metric space this cutoff can also be written explicitly, when and are both nonempty:
The two sets are disjoint and closed, so the denominator never vanishes, and their distance functions are continuous. If is empty, take ; if and is nonempty, take . These choices satisfy the required boundary conditions in the exceptional cases.
The difference vanishes on and outside , and its absolute value is at most one everywhere. Therefore
This is the precise continuous approximation required by both proofs of ergodicity, and also gives .
We first prove invariance, before using any assertion that requires it. Write . Choose a point whose forward orbit has the prescribed convergence against every continuous function; the hypothesis gives a set of such points of full measure. For each continuous function , telescoping gives
because a continuous function on a compact metric space is bounded. Both test functions are continuous, so their averages converge to their prescribed integrals. Hence
for every continuous function . The continuous-test criterion for an invariant measure therefore gives . The points in the hypothesis can now be called generic points for an invariant measure.
For ergodicity, let be a Borel set invariant modulo null sets, and set , . Then almost everywhere. Since invariance preserves null preimages, deleting the union of all iterated preimages of the exceptional null set gives for every on one full-measure set. Thus almost everywhere for every .
Given , the continuous approximation of Borel indicators on a compact metric space, proved in the supporting steps below, supplies with and . By the hypothesis, almost everywhere. Since , the dominated convergence theorem also gives convergence in . Invariance and the triangle inequality give the uniform estimate
Consequently,
Letting and then proves . Explicitly,
so . Every invariant Borel set has measure zero or one, which proves
This proof controls measurable-set averages through a uniform estimate; convergence only against continuous test functions is not silently assumed to include indicator functions.
For a probability measure-preserving system, ergodicity means that every measurable set satisfying modulo null sets has measure zero or one. Equivalently, the only invariant functions are almost everywhere constant. The pointwise ergodic theorem states that, for ,
where is the invariant sigma-algebra; on a probability space the convergence also holds in . For an ergodic transformation, the limit is the constant .
For the Gauss continued-fraction map, the Gauss measure is the probability measure
Its density integrates to one and is bounded above and below by positive constants, so it has the same null sets as Lebesgue measure. Apart from countably many endpoints, the inverse branches are , . For a bounded measurable test function , substitution on each branch gives
Thus is a measure-preserving transformation. Its definition at zero and the rational branch endpoints does not affect these identities.
For ergodicity, we give the full shrinking-cylinder argument. A digit cylinder with first partial quotients is an interval , where and is the identity. The continued fraction convergents give
Here after the initial step, and the recurrence makes tend to infinity at least at the Fibonacci rate. Hence the nested cylinders of every irrational point shrink to that point. The bounded distortion of continued-fraction cylinders is uniform: the ratio of the largest to the smallest value of is at most four. Since ranges between and , the density
has maximum-to-minimum ratio at most eight. Consequently every measurable satisfies
The last inequality uses .
Now suppose modulo null sets and . Choose an irrational density point , and let be its successive digit cylinders. The Lebesgue density theorem applies to these intervals even though they need not be centered: an interval of length containing lies in the centered interval of radius , so its relative complement tends to zero. Comparability of the two densities then gives
But invariance gives almost everywhere for each inverse branch of . Pulling the exceptional null sets through is legitimate since has a positive smooth Jacobian on this fixed branch. Thus the distortion estimate with gives
Letting forces . Therefore the Gauss continued-fraction map is ergodic for the Gauss measure.
For an irrational , the successive continued fraction digits satisfy . The event that a digit equals two is the interval , up to null endpoints. Apply the pointwise ergodic theorem to its indicator:
This holds for almost every with respect to both the Gauss measure and Lebesgue measure, since they have the same null sets.
Use the convention that level counts the nonzero shifts, hence an arithmetic progression of length . The SZ property at level means that every measurable set of positive measure satisfies
This is the averaged multiple-recurrence property, stronger than merely finding one positive intersection. Shifting the indexing convention from terms to terms does not change the all-levels assertion. We work with probability measure-preserving systems.
Here is the Furstenberg correspondence principle construction proving the Szemerédi theorem. Let have positive upper density . Put equal to the indicator of on the positive coordinates and zero elsewhere, and let be the left shift. The binary full shift is a compact metric space in the product topology, and is a homeomorphism. Choose attaining the upper density and define the empirical measures
We use the standard weak compactness fact that every sequence of Borel probability measures on a compact metric space has a weakly convergent subsequence, where weak convergence means convergence of integrals of continuous functions. Pass to a subsequence with limit . For every continuous , telescoping gives
Passing to the limit proves . The cylinder set is clopen, so its indicator is continuous and
If every measure-preserving system has the SZ property at level , there is some for which , where . This is also a clopen cylinder set, so . For some sufficiently large , at least one term of the empirical average is positive: for some . Unwinding the coordinates gives
Therefore contains an arithmetic progression of length with positive common difference. Having the SZ property at every level gives the infinite-density form of the Szemerédi theorem.
The finite uniform form follows from the same argument. If, for fixed and , arbitrarily large admitted sets with and no such arithmetic progression, replace above by the indicator of , extended by zero outside that interval. Their empirical measures still have an invariant weak limit with . For every fixed positive , however, , since a point contributing to that cylinder would be an actual progression in . Thus for all positive , contradicting the SZ property. Consequently a sufficiently large depending only on forces the required progression.
A compact measure-preserving system is one for which every observable has relatively compact forward orbit under the Koopman operator . Equivalently, each of these orbits is totally bounded in the global norm; compactness refers to observables, not just to the underlying topological space. For an invertible system it is usual to use all integer times. One may also define compactness by density of almost periodic observables: the two definitions coincide because , so a finite net for an approximating observable is a finite net, with slightly larger radius, for .
To prove the SZ property, fix with , put , and choose
Compactness gives a finite -net for its orbit with centers , where . Let . For every integer , some center satisfies . Since is an isometry, cancellation gives
Thus the return set meets every interval for . In particular, is a syndetic set with lower density at least . This proves the needed syndetic near returns of an almost periodic observable without any assumption of ergodicity.
For , telescoping and the isometry property give
As is an indicator, . The union bound therefore gives
All other multiple-intersection measures are nonnegative. Averaging over the bounded-gap set yields
Thus every compact measure-preserving system has the SZ property at every level, including nonergodic systems and, under the forward-orbit definition, noninvertible transformations.
Use probability measure-preserving systems: is measurable and . Its invariant sigma-algebra is
Ergodicity means that every invariant measurable set has measure zero or one:
Equivalently, every invariant measurable function is constant almost everywhere. Indeed all rational sublevel sets of a real invariant function belong to , so the zero-or-one alternative forces its distribution to concentrate at one value. Apply this to real and imaginary parts for complex functions. Conversely, an invariant set has an invariant indicator, so the function criterion implies the set criterion. This is the invariant-function characterization of ergodicity.
For an irrational rotation of the circle, with , an invariant has Fourier coefficients satisfying
For the multiplier is nonzero. Completeness of the Fourier series basis therefore makes constant. Apply this to an invariant indicator to prove the irrational rotation is ergodic. A second example is the cyclic permutation of points with equal masses: a nonempty invariant set contains the whole cycle and hence has measure one.
The identity on with Lebesgue measure is not ergodic, since is invariant and has measure . Nor is rotation by on the circle: the set
is invariant and has measure . These examples also show why preservation of measure does not imply ergodicity.
Let be the Koopman operator, , and set
The Von Neumann mean ergodic theorem says that, for , these Cesàro averages converge in to the orthogonal projection onto the invariant functions:
For an invertible system, is a unitary operator. The Hilbert-space theorem states more generally that the Cesàro averages of any unitary operator converge strongly to its fixed-space projection.
Here is the requested invertible-case proof. Since ,
Thus the orthogonal decomposition for unitary ergodic averages is
On the first summand, . On a vector , telescoping gives
Because , approximation extends this convergence to the closure of the range. The orthogonal decomposition now proves the theorem for every . An invariant function is -measurable, and an -measurable function is invariant by approximation with indicators; hence the projection is the stated conditional expectation.
In an ergodic system, invariant functions are constants by the criterion already proved. Their orthogonal projection is obtained by taking the inner product with , whose norm is one. Therefore the ergodic limit is the space average:
The Birkhoff ergodic theorem states that, for ,
The limit is invariant, integrable, and has the same integral as . On a probability space convergence also holds in . In an ergodic system the limit is almost everywhere.
For the topological sharpening, take a continuous map on a compact metric space. Unique ergodicity means there is exactly one invariant Borel probability measure . The uniform ergodic convergence for uniquely ergodic systems says that every continuous real or complex satisfies
Thus convergence holds at every point and uniformly in that point. The continuity, compactness and continuous-observable hypotheses belong to this theorem; it is not a claim of uniform convergence for all measurable functions.
For a probability measure-preserving system, weak mixing is the vanishing of averaged absolute correlation discrepancies:
By approximation with simple functions and the Cauchy-Schwarz inequality, this is equivalent to
Here , and . Absolute values are part of the definition: signed Cesaro convergence of a sequence of these correlations alone expresses ergodicity, not weak mixing.
Suppose the product system is ergodic. Fix a mean-zero and any . On the product take
These belong to , and . The mean ergodic theorem on the product, followed by pairing with , gives
The Cauchy-Schwarz inequality in bounds the averaged absolute correlation by the square root of this quantity. Subtract the mean from a general to obtain the definition above. This proves product ergodicity implies weak mixing, through the square-correlation proof of weak mixing from product ergodicity.
For the multiple averages, write and use the following Van der Corput lemma. If is bounded in a Hilbert space, every correlation average
exists, and , then in norm. One way to see the estimate is to replace by ; for fixed the change in its long average tends to zero. The Cauchy-Schwarz inequality and expansion of the squared block norm give
Let . This proves the auxiliary implication needed here.
A weak mixing system is ergodic: a mean-zero invariant would have the nonvanishing correlation . Every positive power is also weak mixing, since for nonnegative correlation discrepancies ,
This part of the stability of weak mixing under powers and products will control the induction.
In fact the stronger arithmetic-progression multiple averages under weak mixing holds:
For this is the mean ergodic theorem and ergodicity. Suppose it is known for and first assume . Put . For fixed , set
Using invariance of the integral to remove the common gives
This identity does not require an inverse of . Apply the induction hypothesis to the factors and pair their limit with . It follows that
The average in of the right side tends to zero because is weak mixing and . The Van der Corput lemma proves the zero limit. For general , split it into and its constant mean; the first term has the zero limit just proved, and the other term is times the induction average for . This completes the induction.
Pair this result with the real bounded . For
the requested Cesaro limit is
To obtain convergence in density of a sequence, we also need the product system to be weak mixing. For simple tensors, its correlations are products of single-system correlations. If and in averaged absolute discrepancy and both sequences are bounded, then
so the product discrepancy has zero average. Finite sums of tensors are dense in , and the Cauchy-Schwarz inequality extends the conclusion to arbitrary functions. Hence is weak mixing.
Apply the multiple-average result to on that product. Because the original are real,
Together with the first-moment limit this gives
The mean-square criterion for convergence in density now yields, for every ,
Therefore the density convergence of multiple weak-mixing correlations gives the same answer:
The second-moment argument is essential; the signed Cesaro limit alone would not imply this conclusion.
For an invertible probability measure-preserving system, an almost periodic observable is an whose Koopman operator orbit is totally bounded:
Equivalently, for every it has a finite -net in . For a noninvertible system use the forward orbit .
A factor map satisfies and almost everywhere. Disintegrate over this map and write
for the conditional L2 norm. A relatively almost periodic observable is an such that, for every , there are finitely many with
The center may depend on both and ; the finite list itself does not. Since is countable, the exceptional null sets can be combined. A compact extension of a measure-preserving system is a factor map for which these relatively almost periodic observables are dense in . This density definition should not be replaced by a claim that every observable already satisfies the uniform fiberwise finite-net condition.
For a proper factor map example, let carry independent fair coordinates and the two-sided Bernoulli shift , with . Set
using the product of the Bernoulli measure and the equal two-point measure. These are invertible measure-preserving systems, and the projection preserves measure and intertwines the transformations, so it is a two-to-one factor map.
Take . Then . Distinct coordinates are independent and have mean zero and variance one, hence
Its orbit has no finite sufficiently small net. Thus the Bernoulli coordinate observable is not almost periodic in global . In contrast, the conditional measure over is the equal measure on , and is constant on that fiber with value . The two centers give zero error on every fiber for every . Therefore is a relatively almost periodic observable for this map.
The example is also a finite-fiber compact extension. For a bounded observable on , each orbit value on a fiber is a pair in a fixed bounded subset of , which has a finite net. Its centers can be chosen as functions constant in , with prescribed values on the two labels. Bounded observables are dense in , so the extension is compact. The Bernoulli shift is mixing: cylinder events involving disjoint coordinate sets are independent for all sufficiently large shifts, and approximation by cylinder events extends this to arbitrary events. Thus and are ergodic. If an invariant function on is written , invariance gives and . The first function is constant; the second is -invariant and hence constant, and then its sign equation forces zero. This proves that the example's source is also ergodic.
For the positive-subset assertion, interpret the printed as , or as an invariant full-measure domain of the factor map; the prime is otherwise undefined and does not change the measure-theoretic argument.
First establish conditional norm covariance under a factor map. Invariance of the measures and uniqueness in the disintegration theorem for a probability measure give
on common full-measure sets. For the first equality, is another disintegration over the same fibers: it is supported on and integrates to . Uniqueness identifies it with , and iteration gives the norm formula.
Two elementary facts explain localization of relative almost periodicity to base sets. If is a relatively almost periodic observable and is measurable, then is also relatively almost periodic. Indeed, on a fiber,
When that indicator is one, use a center for ; when it is zero, use the additional center . Secondly, relative almost periodicity is closed under convergence in the uniform conditional L2 norm
If this norm of tends to zero, covariance bounds the corresponding error between every pair of orbit values by the same number. A finite -net for one sufficiently close is consequently an -net for .
Now take . The compact extension supplies relatively almost periodic observables with
Put . The Tonelli theorem gives
Hence for -almost every . Since , choose for which
has positive measure. The Egorov theorem gives a measurable of positive measure on which uniformly. Define
Then
For , the localization fact makes every relatively almost periodic, while
The conditional-norm closure fact therefore proves is a relatively almost periodic observable, with and positive measure. This is the positive-measure almost periodic indicator in a compact extension. No ergodic component or uncountable intersection of exceptional sets is needed in this construction; it in fact works under the same disintegration and compactness hypotheses without using the assumed ergodicity.
For a probability distribution on , take the canonical sample space , the product of the Borel -algebras, and the product measure . The coordinate functions have joint cylinder probabilities
The Kolmogorov extension theorem supplies this measure from the consistent finite-dimensional distributions, and uniqueness follows from the generating cylinder sets. Thus the coordinates are independent and identically distributed; the same construction works for a given standard Borel state space.
The one-sided Bernoulli shift is . It is measurable and preserves all cylinder probabilities. The uniqueness theorem for probability measures extends this to the product -algebra, proving measure preservation.
To prove ergodicity, let be invariant modulo null sets and approximate it in probability by an event depending on only the first coordinates, with . Such approximations follow from the fact that finite-coordinate cylinder events generate the product -algebra. For , independence gives . Invariance and measure preservation give
Also . Letting proves , so . This is exactly ergodicity.
Poisson arrivals see time averages Created 2026-09-24 Updated 2026-10-14
Poisson arrivals see time averages says that an exogenous, homogeneous Poisson process samples prearrival states with the same long-run frequencies as observation over time, provided the observed system cannot anticipate future arrivals and the relevant time averages exist. The system may depend on all past arrivals: an ordinary queue does. The required independence is between future increments of the Poisson process and the history available to the system, including its service information. Assume measurable state observations with ordinary piecewise continuous functions as sample paths; write for the state immediately before time .
For a state set , let count offered arrivals by time and let be their times. With a positive finite constant arrival rate, the PASTA equality is
almost surely when the time limit exists. The indicator functions count visits on the left and time spent on the right. For a stationary process with ergodicity, the right side is the stationary probability of . It matters that the arriving customer has not yet changed the observed state.
In the M/M/1 utilization and delay example, server utilization is . Thus PASTA says that 80 percent of offered arrivals find the server busy and 20 percent find the whole queue empty. If an offered arrival seeing population is admitted with conditional probability , using fresh randomness independent of that state, the accepted fraction is , where is the stationary distribution. Multiplying this fraction by the offered rate gives the admitted queue throughput when every admitted customer eventually completes service. Accepted arrivals are selected by state and need not themselves form a Poisson process or see the unconditioned time frequencies.
Arbitrary arrivals need not see time averages. Suppose a customer arrives every two minutes and its service time is exactly one minute, with one server and no other customers. The server is busy half the time, but every arrival finds it idle. A uniformly random initial phase can even make the occupancy a stationary process, without repairing the disagreement. Nevertheless Little's law still gives , with rate measured in customers per minute. This separates Little's law from PASTA.
Ronald W. Wolff's 1982 paper gave a general PASTA theorem under nonanticipation, rather than requiring the observed system to be independent of its driving Poisson process. Its Theorem 1 compares the two long-run fractions; stationarity is a way to establish convergence, not the central sampling condition. [ref]. [ref].
Queue population 2026-10-14
The population of a queue counts every admitted customer still inside the selected system at time , including customers receiving service when the server is inside that boundary. Its long-run time average is . The waiting population excludes customers currently receiving service, and its long-run time average is . For a single server that serves whenever work is present,
where is the server utilization, assuming at most one customer receives service at a time. In a stationary process with ergodicity and finite expected values, these time averages equal and . The workload of a queue instead counts unfinished work, so two queues with the same population can have very different remaining workloads.