Past exam of the options II course of the University of Oxford / 2013 / M4

A Markov chain is irreducible if every state communicates with every other: for any $i,j$ there is an integer $n\ge 0$ with $p_{ij}^{(n)}>0$. Thus any state can be reached from any other in finitely many steps with positive probability.

Every state reaches state one by deterministic downward steps. From one, the chain can reach a given state $j$ exactly when it can jump to some $k\ge j$ with ${\alpha }_k>0$ and then count down to $j$. ThereforeIf the support is bounded by $M$, states above $M$ cannot be reached from one, so the chain on the stated infinite state space is not irreducible. Its smaller closed class can still have a stationary distribution, as in part (e).

After a jump from one to $k$, the first return to one occurs exactly $k$ steps later, with probability ${\alpha }_k$. Later return times are sums of such positive-support values. Thus their greatest common divisor isIn an irreducible chain this is the common period of every state. Hence the chain is aperiodic exactly when $d=1$. A self-loop at one is sufficient but not necessary; support at two and three also has greatest common divisor one.

The first return time ${\tau }_1$ has distribution ${\mathbb{P}}_1({\tau }_1=k)={\alpha }_k$. It is finite almost surely, so state one is recurrent. It is positive recurrent precisely when its mean return time is finite:By irreducibility the same recurrence class contains every state. Thus the condition is finiteness of the expected value of the positive-integer distribution $({\alpha }_k)$. If that mean is infinite, the chain is null recurrent, not transient.

The stationary balance equations on the closed class $\{1,\dots ,M\}$ are ${\pi }_j={\pi }_{j+1}+{\pi }_1/M$, with ${\pi }_{M+1}=0$. Working backward gives ${\pi }_j={\pi }_1(M-j+1)/M$.

Normalization yields $1={\pi }_1(M+1)/2$, soThese probabilities sum to one and satisfy each balance equation. In particular ${\pi }_1=2/(M+1)$ is the reciprocal of the mean return time $(M+1)/2$, consistent with part (d).

Write a first return time as $2J$, where $\mathbb{P}(J=j)=2^{-j}$ for $j\ge 1$. It has mean $2\cdot 2=4$ and period two, so the stationary probability at one is $1/4$. The periodic convergence theorem would give limits $1/2$ along even times and zero along odd times.

Here one can obtain them directly. Put $u_n={\mathbb{P}}_1(X_{2n}=1)$, with $u_0=1$. Splitting according to the first return gives the renewal equationInduction gives $u_n=1/2$ for every $n\ge 1$: the right side is $2^{-n}+\frac{1}{2}{\sum }_{j=1}^{n-1}{2}^{-j}=1/2$. Odd-time returns are impossible because every return interval is even. Therefore the exact set of limit points isThe sequence does not converge to the stationary mass $1/4$, precisely because it is periodic.