Past exam of the options II course of the University of Oxford / 2010 / M / M5

A time-homogeneous Markov chain on a countable state space $S$ is a sequence of $S$-valued random variables for which, for every positive-probability finite history,Thus the conditional next-step law depends only on the current state, and the transition probabilities do not depend on time.

The transition matrix is $P=(P_{ij}{)}_{i,j\in S}$, where $P_{ij}=P(X_{n+1}=j\mid X_n=i)$. Its entries are nonnegative and each row sums to one. On states absent from a particular initial law, these probabilities specify the same chain when started from that state.

For $n=0$, the transition probabilities are the entries of the identity $P^0$. Suppose the formula holds for $n$. Conditioning on $X_n=k$ and using the Markov property givesThe countable sums have nonnegative terms, so they are justified by the law of total probability. Induction proves

Conditioning on the initial state givesThus the row-vector distribution at time $n$ is $\lambda P^n$.

If $\lambda P=\lambda$, induction gives $\lambda P^n=\lambda$ for every $n\ge 0$. Part (c) then makes the distribution of every $X_n$ equal to $\lambda$. Such a probability row vector is a stationary distribution.

For $i\ge 1$, the stationarity equation iswhich rearranges to the stated recurrence. At zero,Define the adjacent probability current $J_i=a{\pi }_i-b{\pi }_{i+1}$. The recurrence says $J_i=J_{i-1}$, and the boundary equation says $J_0=0$. Hence ${\pi }_{i+1}=(a/b){\pi }_i$ for all $i$. Since $\rho =a/b<1$, normalization givesThe adjacent balance equations also verify stationarity directly. If $a=0$, this formula means the point mass at zero; otherwise the birth-death chain is irreducible and this is its unique stationary probability distribution.

Let $T_0^{+}=\inf \{n\ge 1:X_n=0\}$. For $a>0$, the chain is irreducible and its stationary probability distribution makes it positive recurrent. The mean-return-time identity givesThis counts a self-loop at zero as a return after one time unit, as “next zero” requires. Equivalently, with probability $1-a$ that immediate return occurs; with probability $a$ the chain moves to one and then has mean hitting time $1/(b-a)$ back to zero, giving $1+a/(b-a)$. If $a=0$, the return is deterministically at time one and the displayed formula still equals one.