Positive recurrent Markov chain

An irreducible Markov chain is positive recurrent when its mean return times are finite. For a nonexplosive Markov chain on a countable state space, this is equivalent to the existence of a stationary distribution on its irreducible class. For continuous time, a return cycle includes departure and return, rather than counting immediate residence as a return.

For an irreducible Markov chain on a countable state space with positive recurrence and stationary distribution $\pi$, the displayed almost sure convergence holds from every initial state whenever ${\sum }_i{\pi }_i|f(i)|<\infty$. In particular, the empirical fraction of visits to $i$ converges to ${\pi }_i$. No aperiodicity is needed. Regeneration at successive returns to a state reduces the result to the strong law of large numbers for independent excursion lengths and rewards. This extends the finite-state Markov chain ergodic theorem.