Past exam of the mathematics course of the University of Cambridge / 2017 / ii / Paper 1 / 27K (Applied Probability)

A time-homogeneous continuous-time Markov chain on a countable state space is a stochastic process with the Markov property: conditional on the current state, the law of its future depends on neither its past nor the current time. In the usual conservative, nonexplosive setting its sample paths are right-continuous step functions. Its Q-matrix $Q=(q_{ij})$ has $q_{ij}\ge 0$ for $j\ne i$, $q_i=-q_{ii}={\sum }_{j\ne i}{q}_{ij}<\infty$, and infinitesimal transition probabilities ${\mathbb{P}}_i(X(h)=j)=q_{ij}h+o(h)$ for $j\ne i$.

On entering $i$, the holding time has distribution $\mathrm{Exp}(q_i)$; independently, the next state is $j$ with probability $q_{ij}/q_i$. The successive states form the jump chain, whose transition probabilities areIf $q_i=0$, the state is absorbing, its holding time is infinite, and the jump-chain convention is $p_{ii}=1$. Nonexplosion ensures this construction defines the process for all finite times rather than accumulating infinitely many jumps in finite time.

Let $r_i$ be the probability of returning to $i$ after its first departure. Transience means $r_i<1$; in particular $q_i>0$. By the Strong Markov property, the number $N$ of visits, including the initial visit, has the geometric distribution $\mathbb{P}(N=n)=(1-r_i)r_i^{n-1}$, $n\ge 1$. The holding times of these visits are independent $\mathrm{Exp}(q_i)$ variables and are independent of the jump-chain return decisions.

For the total occupation time of a continuous-time Markov chain $T_i$ and $s\ge 0$, summing over $N$ givesUniqueness of the Laplace transform identifiesStarting elsewhere adds an atom at zero if the state might never be hit; the purely exponential conclusion uses the specified initial state $i$.

Write $a=\mu /\lambda <1$. The embedded birth-death chain moves upward with probability $\lambda /(\lambda +\mu )$ and downward with probability $\mu /(\lambda +\mu )$ away from zero. The gambler's ruin formula, obtained by solving its two-point difference equation on $\{0,\dots ,N\}$ and then letting $N\to \infty$, gives ${\mathbb{P}}_k(\text{hit }0)=a^k$. Translation gives return probability $a$ from $r+1$ to $r$. From $r-1$ the chain hits $r$ almost surely: before that hit it remains in a finite reflecting interval, from every state of which a uniformly positive finite-step chance of hitting $r$ is available. This also proves that starting at zero it hits each $r\ge 1$ almost surely.

After departure from $r\ge 1$ the return probability is thereforeThe holding rate is $q_r=\lambda +\mu$, so the exponential occupation-time parameter is $q_r(1-r_r)=\lambda -\mu$. Applying the Strong Markov property at the almost-sure first hit givesAt zero the holding rate is $\lambda$ and return probability after departure is $a$, again giving parameter $\lambda -\mu$. From initial state $k$, the expected occupation time at zero is $a^k/(\lambda -\mu )$, since the time is zero unless zero is reached. Averaging over the initial law with probability generating function $G$ yieldsFor $\mu =0$, the interpretation is $a^0=1$, $a^k=0$ for $k>0$, and $G(0)=\mathbb{P}(X(0)=0)$; the same formulas hold.