Past exam of the Probability, statistics and options II course of the University of Oxford / 2010 / M / M5

A time-homogeneous Markov chain on a countable state space satisfies, whenever the conditioning event has positive probability,where the last probabilities do not depend on $n$. Thus the present state contains all the past information relevant to the next transition. Time homogeneity is the convention used here, since the problem specifies a single transition matrix.

The transition matrix is $P=(p_{ij}{)}_{i,j\in S}$, where $p_{ij}=P(X_{n+1}=j\mid X_n=i)$. It is row-stochastic: $p_{ij}\ge 0$ and ${\sum }_j{p}_{ij}=1$ for each $i$. Initial states of zero probability can be specified by the chain's transition mechanism, so the matrix describes the chain from any starting state.

At $n=0$ the transition probability is ${\delta }_{ij}=(P^0{)}_{ij}$. If the assertion holds at $n$, condition on $X_n=k$ and use the Markov property:Induction proves $\boxed{P(X_n=j\mid X_0=i)=(P^n{)}_{ij}}$. These countable sums contain nonnegative terms and are justified by total probability.

Conditioning on the initial state givesThe row-vector convention is important: the initial distribution multiplies the transition matrix on the left.

If $\lambda P=\lambda$, induction gives $\lambda P^n=\lambda$ for all $n\ge 0$. By (c), the marginal distribution at time $n$ is consequently $\boxed{\lambda }$. Such a probability vector is a stationary distribution.

The stationarity equation at an interior state collects arrivals from its two neighbors and itself:This rearranges to the required recurrence. At zero, stationarity gives ${\pi }_0=(1-a){\pi }_0+b{\pi }_1$, hence $\boxed{b{\pi }_1=a{\pi }_0}$.

The recurrence says the flux $a{\pi }_{i-1}-b{\pi }_i$ equals $a{\pi }_i-b{\pi }_{i+1}$. The boundary equation makes that flux zero, so ${\pi }_i=(a/b{)}^i{\pi }_0$. Since $0\le a/b<1$, normalization givesWhen $a=0$ this means ${\pi }_0=1$ and ${\pi }_i=0$ for $i>0$. The probability assumptions also require $a,b\ge 0$, $a+b\le 1$, as implicit in the transition rules.

For $a>0$ the chain is irreducible, and the normalized stationary distribution in (i) makes it positive recurrent. The mean-return-time formula givesHere $T_0^{+}=\inf \{n\ge 1:X_n=0\}$ includes a return at time one when the chain stays at zero. This convention is essential. Equivalently the mean hitting time of zero from one is $1/(b-a)$, so a first-step decomposition gives $1+a/(b-a)$, the same result. For $a=0$ the population remains zero and the next zero is at time one, also agreeing with the formula.