Past exam of the mathematics course of the University of Cambridge / 2013 / ib / Paper 2 / 20H / i / Solution

Let $T_j=\inf \{n\ge 1:X_n=j\}$. By the Strong Markov property, the probability of hitting $j$ and subsequently returning to $i$ is $f_{ij}{f}_{ji}$. This event implies a return to $i$, so $f_{ii}\ge f_{ij}{f}_{ji}$. For $i=j$, interpret the event as two successive returns.

Set $T_i^{(0)}=0$ and let $T_i^{(r)}$ be the $r$th successive return. Repeated use of the Strong Markov property gives ${\mathbb{P}}_i(T_i^{(r)}<\infty )=f_{ii}^r$. The total number $N_i$ of visits, including time zero, is both ${\sum }_{n\ge 0}{1}_{\{X_n=i\}}$ and ${\sum }_{r\ge 0}{1}_{\{T_i^{(r)}<\infty \}}$. By Tonelli theorem,The identity is valid with value $+\infty$. In particular, divergence of the return-probability sum is equivalent to Markov-chain recurrence.