Past exam of the mathematics course of the University of Cambridge / 2021 / ib / Paper 3 / 8H (Markov Chains)

States $i$ and $j$ communicate when each is accessible from the other: there exist $m,n\ge 0$ withThis is an equivalence relation, and its equivalence classes are the communicating classes. A class is closed when no positive-probability transition leaves it.

Reading the directed edges with positive transition probability, the communicating classes areState $3$ can move to $2$ or $4$, and state $4$ can move to $2$ or $5$, so their singleton classes are not closed. Every transition from $1,2,5,$ or $6$ remains within $\{1,2,5,6\}$, so

Let $h_i={\mathbb{E}}_i{T}_6$. Then $h_6=0$, and first-step analysis on the closed class givesSolving these linear equations gives