Past exam of the mathematics course of the University of Cambridge / 2016 / ib / Paper 2 / 20H / b / iii / Solution

The only closed communicating class is $\{b,d\}$; the class $\{a,c\}$ is open and transient. Its one-step probability of staying inside is at most $2/3$, so if $\tau$ is the entrance time into $\{b,d\}$, then $\mathbb{P}(\tau >n)\le (2/3{)}^n$ for an initial state in $\{a,c\}$. Hence entrance occurs almost surely in finite time.

On $\{b,d\}$ it is an irreducible aperiodic Markov chain, because both states have positive self-loop probabilities. The finite irreducible aperiodic countable-state Markov chain convergence theorem says that its transition probabilities converge to its unique stationary distribution. The balance equation ${\pi }_b(3/4)={\pi }_d(2/3)$ with ${\pi }_b+{\pi }_d=1$ gives ${\pi }_b=8/17$ and ${\pi }_d=9/17$.

This limit also holds from $a,c$: condition on the finite entrance time and entrance state. For $\tau \le M$, convergence in the closed class applies to a finite sum; the remaining probability is bounded by $\mathbb{P}(\tau >M)$, which tends to zero as $M\to \infty$. Thus in the order $a,b,c,d$,