Past exam of the options I course of the University of Oxford / 2011 / M / 2

Condition on the intermediate state. Starting at $+1$, a return to $+1$ after two transitions occurs through $-1$ with probability $q(1-p)$ or through $+1$ with probability $(1-q{)}^2$. These alternatives are disjoint, so

A stationary distribution is a probability row vector $\pi$ satisfying $\pi P=\pi$. Its coordinates obey ${\pi }_{-}(1-p)={\pi }_{+}q$ and ${\pi }_{-}+{\pi }_{+}=1$. ThusBecause both transitions between distinct states have positive probability, the finite Markov chain is irreducible; its positive self-transition probabilities also make it aperiodic.

The finite irreducible aperiodic Markov chain converges to its stationary distribution, regardless of its initial law. Therefore $\mathbb{E}[X_j]\to {\pi }_{+}-{\pi }_{-}$. Taking the arithmetic means of this convergent sequence givesThis uses linearity of expectation and convergence of Cesàro means; it does not require the variables $X_j$ to be independent.

With probability $2/3$, the chain starts at $+1$ and $T=1$. Conditional on starting at $-1$, it takes a geometric number $G$ of transitions to first reach $+1$, with success probability $1-p$, so $T=1+G$. The probability generating function is thereforeThe parameter $q$ is absent because the process stops at its first visit to $+1$. At $s=1$ the expression equals one, confirming eventual hitting with probability one.