Past exam of the mathematics course of the University of Cambridge / 2019 / ii / Paper 1 / 28K (Applied Probability)

Conditional on $Y_m=i$, the holding time $V_m=U_m/g(i)$ has an exponential distribution of rate $g(i)$, and after that holding time the next state is $j$ with transition probability $p_{ij}$. The memoryless property of exponential random variables and the Markov property of $(Y_m)$ therefore make $(X_t)$ a continuous-time Markov chain.

Moreover $g(i)<1/\epsilon$, so $V_m\ge \epsilon U_m$. The strong law of large numbers gives ${\sum }_m{U}_m=\infty$ almost surely, hence $J_m\to \infty$ and the process is nonexplosive. Its Q-matrix isEquivalently, if $D_g$ is the diagonal matrix with entries $g(i)$, then

A continuous-time Markov chain is irreducible if every state can reach every other state with positive probability. A state $i$ is recurrent if, after leaving $i$, the chain returns to $i$ with probability one; an irreducible chain is recurrent if one, and hence every, state is recurrent. A recurrent state is positive recurrent if its expected continuous first return time is finite, and an irreducible chain is positive recurrent when all its states are.

The jump chain of $(X_t)$ is exactly $(Y_m)$: from $i$ it visits $j$ next with probability $p_{ij}$. Consequently the two chains have exactly the same directed paths of positive-probability transitions. Thus

Because $J_m\to \infty$ almost surely, no infinite portion of the jump chain is hidden inside a finite time interval. After either chain leaves a state $i$, the event that it ever returns is therefore the same event for both chains. Hence irreducibility and recurrence pass between a CTMC and its jump chain gives

Let $\nu$ be an invariant distribution for the continuous-time Markov chain. Using $Q=D_g(P-I)$, its stationarity equation $\nu Q=0$ becomesThus the measure with weights ${\nu }_{i}g(i)$ is invariant for the irreducible positive-recurrent jump chain. Its invariant distribution $\pi$ is unique, so ${\nu }_{i}g(i)=c{\pi }_i$. Normalization gives the invariant-measure transfer between a jump chain and a CTMC formulaThe denominator is finite and nonzero because $\epsilon <g(j)<1/\epsilon$. Thus $\nu$ is a probability distribution and $(X_t)$ is positive recurrent.