Past exam of the mathematics course of the University of Cambridge / 2016 / ib / Paper 2 / 20H / b

The states $b,d$ form a closed communicating class. Set $p_n=\mathbb{P}(X_n=b\mid X_0=d)$. Within this class,Its fixed point is $8/17$. Subtracting that fixed point and iterating gives the exact transition probability

Let $h$ be the probability of ever hitting $c$ starting at $a$. A first-step analysis gives a direct success with probability $1/3$, a return to the same problem with probability $1/3$, and entry into the closed class $\{b,d\}$ with probability $1/3$, after which $c$ cannot be reached. ThereforeThe possibility of infinitely many self-loops has probability zero and introduces no extra solution of this equation.

The only closed communicating class is $\{b,d\}$; the class $\{a,c\}$ is open and transient. Its one-step probability of staying inside is at most $2/3$, so if $\tau$ is the entrance time into $\{b,d\}$, then $\mathbb{P}(\tau >n)\le (2/3{)}^n$ for an initial state in $\{a,c\}$. Hence entrance occurs almost surely in finite time.

On $\{b,d\}$ it is an irreducible aperiodic Markov chain, because both states have positive self-loop probabilities. The finite irreducible aperiodic countable-state Markov chain convergence theorem says that its transition probabilities converge to its unique stationary distribution. The balance equation ${\pi }_b(3/4)={\pi }_d(2/3)$ with ${\pi }_b+{\pi }_d=1$ gives ${\pi }_b=8/17$ and ${\pi }_d=9/17$.

This limit also holds from $a,c$: condition on the finite entrance time and entrance state. For $\tau \le M$, convergence in the closed class applies to a finite sum; the remaining probability is bounded by $\mathbb{P}(\tau >M)$, which tends to zero as $M\to \infty$. Thus in the order $a,b,c,d$,