Past exam of the mathematics course of the University of Cambridge / 2014 / ib / Paper 1 / 20H (Markov Chains)

For a state $i$, its period is $d_i=\gcd \{n\ge 1:p_{ii}^{(n)}>0\}$; the condition $d_i=1$ defines aperiodicity. It is a positive recurrent state if return occurs almost surely and its first strictly positive return time has finite expectation. An ergodic state of a Markov chain is positive recurrent and aperiodic.

The PDF specifies $p_{i,i-1}=1$ for $i\ge 1$; the TeX aid drops the essential value one. Consequently every positive state reaches zero deterministically. From zero, every positive state $j$ is reached in one step with probability $p{q}^{j-1}>0$. Paths through zero connect any pair of states, proving irreducibility.

Let $J$ be the jump out of zero. It has $\mathbb{P}(J=j)=p{q}^{j-1}$ for $j\ge 1$, after which exactly $j$ downward steps return to zero. Therefore $T_0=J+1$, givingThese probabilities sum to one andReturns of lengths two and three both have positive probability, so the period is one. Hence state zero is ergodic, and by irreducibility all states are positive recurrent and have the same period.

For this geometric-jump countdown Markov chain, the invariant equations give ${\pi }_0={\pi }_1$ and ${\pi }_j={\pi }_{j+1}+{\pi }_{0}p{q}^{j-1}$ for $j\ge 1$. Summing the tail, or counting visits in a regeneration cycle, yields ${\pi }_j={\pi }_0{q}^{j-1}$. Normalization givesThis is the stationary distribution of a regenerative countdown chain. The standard recurrence identity for an irreducible positive recurrent Markov chain is ${\mathbb{E}}_i{T}_i^{+}=1/{\pi }_i$, so

The convergence theorem for an irreducible aperiodic positive recurrent Markov chain says ${\mathbb{P}}_i(X_n=j)\to {\pi }_j$ for every pair of states. Apply it to $i=j=0$ and take the complement:Although $p_{00}=0$, the chain is aperiodic because its possible return lengths have greatest common divisor one; a self-loop is sufficient but not necessary for aperiodicity.