Finite Markov chain

For an irreducible finite Markov chain with stationary distribution $\pi$, the fraction of visits to any state $j$ converges almost surely to ${\pi }_j$, for every initial state. More generally, the empirical average of a function of the state converges to its average under $\pi$. Aperiodicity is needed for convergence of the one-time distributions, but not for this empirical-frequency conclusion. Regeneration at returns to a state proves the conclusion by applying the strong law to independent excursion lengths and excursion rewards.

For a real function $f$ on a finite state space, a state minimizing $f$ satisfies $Pf\ge f$, so strictly negative drift cannot hold at every state. Equivalently, telescoping the expected value would give ${\mathbb{E}}_{x}f(X_n)=f(x)-n\delta$, contradicting boundedness of $f$. Constant negative drift is useful before a target is reached; it must be restricted to the states outside that target.