Markov chain ergodic theorem

For an irreducible finite Markov chain with stationary distribution $\pi$, the fraction of visits to any state $j$ converges almost surely to ${\pi }_j$, for every initial state. More generally, the empirical average of a function of the state converges to its average under $\pi$. Aperiodicity is needed for convergence of the one-time distributions, but not for this empirical-frequency conclusion. Regeneration at returns to a state proves the conclusion by applying the strong law to independent excursion lengths and excursion rewards.