Past exam of the options I course of the University of Oxford / 2012 / M / M1

The sequence is a Markov chain when the conditional law of its next state depends on the past only through the current state. Precisely, for each history of positive probability and every $j\in S$,A time-homogeneous chain additionally has transition probabilities independent of $n$, written $P_{ij}=\mathrm{Pr}(X_{n+1}=j\mid X_n=i)$. The displayed single transition matrix in the subsequent parts specifies this time-homogeneous case.

First the entries must be nonnegative: together with the given $a,b\ge 0$, validity of the transition matrix requires $0\le a\le 1$ and $0\le b\le 1/3$. Every row then sums to one.

The only incoming edge to state $2$ is $1\to 2$, of probability $a$. If $a=0$, no other state can reach $2$, so the chain is not irreducible. If $a>0$, all edges in the directed cyclehave positive probability, since $P_{23}=1/3+b>0$, $P_{34}=1$ and $P_{41}=1/2$. Traversing this cycle connects every state to every other state. Therefore

Within the valid parameter rectangle, suppose $a>0$ and $b<1/3$. The chain is then irreducible by part (i). State $1$ has a two-step return $1\to 2\to 1$ and a three-step return $1\to 2\to 4\to 1$, both of positive probability. Its period of a state in a Markov chain therefore divides both two and three, hence equals one. Irreducibility gives the same period to every state, so the chain is aperiodic.

If $b=1/3$ and $a>0$, every transition goes between the two classes $\{1,3\}$ and $\{2,4\}$. Returns must have even length, while $1\to 2\to 1$ has length two. Thus the irreducible chain has period two.

If $a=0$, the closed communicating class $\{1,3,4\}$ alternates between $\{1,3\}$ and $\{4\}$. Its states have period two, using the positive two-step returns through $4$. Consequently the chain is not aperiodic, regardless of how the transient state $2$ is classified. Combining the cases,

Here the valid parameters lie in the irreducible aperiodic range. The finite-state theorem on convergence to an invariant distribution therefore implies convergence to the unique stationary distribution, independently of the initial state.

Solve $\pi P=\pi$ and ${\sum }_i{\pi }_i=1$. The equations for states $1,2,3$ giveHence ${\pi }_3={\pi }_1$ and ${\pi }_4=5{\pi }_1/3$. Normalization gives ${\pi }_1(1+1/2+1+5/3)=1$, soStationarity alone would not guarantee convergence of the one-time distributions; irreducibility and aperiodicity justify that final limit.