Recurrent Markov chain

A state is recurrent when the chain returns to it almost surely after starting there.

An irreducible Markov chain is null recurrent when every state is a null recurrent state: it returns to that state almost surely, but its mean recurrence time is infinite. Such a countable-state chain has no stationary distribution.

An irreducible Markov chain is positive recurrent when its mean return times are finite. For a nonexplosive Markov chain on a countable state space, this is equivalent to the existence of a stationary distribution on its irreducible class. For continuous time, a return cycle includes departure and return, rather than counting immediate residence as a return.

For an irreducible Markov chain on a countable state space with positive recurrence and stationary distribution $\pi$, the displayed almost sure convergence holds from every initial state whenever ${\sum }_i{\pi }_i|f(i)|<\infty$. In particular, the empirical fraction of visits to $i$ converges to ${\pi }_i$. No aperiodicity is needed. Regeneration at successive returns to a state reduces the result to the strong law of large numbers for independent excursion lengths and rewards. This extends the finite-state Markov chain ergodic theorem.

A recurrent state is positive recurrent when its expected return time is finite.

In the standard discrete-time state terminology, an ergodic state is a positive recurrent state with aperiodicity. Return occurs almost surely, the mean strictly positive return time is finite, and the greatest common divisor of possible return lengths is one. An irreducible chain shares these properties among all its states.

A recurrent state is null recurrent when its expected return time is infinite.

A state is recurrent exactly when the sum over time of its return probabilities diverges.