Past exam of the mathematics course of the University of Cambridge / 2023 / ib / Paper 4 / 7H (Markov Chains)

Let $g(0)=0$. First-step analysis givesand, because state $3$ moves to $2$ or remains at $3$ with equal probabilities,The last equation gives $g(3)=2+g(2)$. Substitution into the second gives $g(2)=4+g(1)$, and the first then gives $g(1)=3+g(1)/2$. HenceThese are the expected hitting times of the absorbing state.

Let $h(i)$ be the probability of reaching state $3$ before state $0$. The boundary values are $h(0)=0$ and $h(3)=1$. First-step analysis at states $1$ and $2$ givesTherefore $h(1)=h(1)/4+1/4$, so the requested hitting probability is

Let $u(i)$ be the expected number of visits to state $3$ before absorption, counting the present state when $i=3$. Its reward equations areThe last equation gives $u(3)=2+u(2)$, so the second nontrivial equation gives $u(2)=2+u(1)$. Hence $u(1)=1+u(1)/2$, andThus the expected occupation count is two; together with the previous parts, this is the four-state reflecting absorbing random walk calculation.