 Countable-state Markov chain ergodic theorem

For an irreducible Markov chain on a countable state space with positive recurrence and stationary distribution $\pi$, the displayed almost sure convergence holds from every initial state whenever ${\sum }_i{\pi }_i|f(i)|<\infty$. In particular, the empirical fraction of visits to $i$ converges to ${\pi }_i$. No aperiodicity is needed. Regeneration at successive returns to a state reduces the result to the strong law of large numbers for independent excursion lengths and rewards. This extends the finite-state Markov chain ergodic theorem.