Birth-death chain

A birth-death chain is a discrete-time Markov chain on nonnegative integers that can move only to the same state or a neighbouring state.

In a positive recurrent birth-death-immigration jump chain, let $I_n$ and $A_n$ count immigration events and upward steps before time $n$. The centered immigration indicators form bounded martingale differences; the strong law for martingales with bounded increments and the countable-state Markov chain ergodic theorem give $I_n/n\to {\sum }_k{\pi }_k\theta /(k+\theta )$ almost surely. The analogous argument for upward steps gives $A_n/n\to {\sum }_k{\pi }_k{u}_k$. Summing detailed balance gives ${\sum }_k{\pi }_k{u}_k={\sum }_k{\pi }_k{d}_k=1/2$, since every step is upward or downward. Therefore $I_n/A_n$ converges to the displayed fraction. For this chain the explicit stationary distribution reduces the fraction to $1-p/(1-p)$.

Here $0<p<1$, $\theta >0$, and the upward step is either a birth or immigration, the latter having conditional probability $\theta /(k+\theta )$. At zero immigration occurs with probability one. Put $q=1-p$, $a=\theta /p$ and $r=p/q$. The stationary weights of a reflected birth-death chain areTheir successive ratio tends to $r$, so they are summable for $p<1/2$. For $p\ge 1/2$, $u_k>1/2>d_{k+1}$ makes the weights strictly increasing, ruling out a stationary distribution. Thus positive recurrence holds exactly for $p<1/2$. In that case the binomial series gives ${\sum }_k{w}_k=2(1-r{)}^{-a-1}$, andEvery step reverses parity and a two-step return has positive probability, so the period of a state in a Markov chain is two. Starting at zero, $\mathbb{P}(X_{2n}=0)\to 2{\pi }_0=(1-r{)}^{a+1}$, whereas $\mathbb{P}(X_{2n+1}=0)=0$.

Consider an irreducible Markov chain on the nonnegative integers with up-step probabilities $u_k>0$ and down-step probabilities $d_k>0$ for $k\ge 1$, reflected at zero. The displayed weights satisfy detailed balance. Every stationary distribution is proportional to them: the stationary equation at zero sets the edge flow ${\pi }_0{u}_0-{\pi }_1{d}_1$ to zero, and the remaining stationary equations make the flow constant along all edges. Hence positive recurrence is equivalent to ${\sum }_k{w}_k<\infty$, and then ${\pi }_k=w_k/{\sum }_j{w}_j$. This also allows holding probabilities of a Markov chain.

A strictly increasing sequence $s_i$ for which $s(X_t)$ is a martingale along the birth-death chain before hitting its boundary. With upward and downward probabilities $p_i,q_i$, harmonicity means $p_i(s_{i+1}-s_i)=q_i(s_i-s_{i-1})$. This coordinate is useful for hitting probabilities and optimal stopping.

For a nearest-neighbour Markov chain on $\{L,\dots ,U\}$ with positive right-step probabilities in its interior, let $m=U-L$ and $p_{\ast }>0$ be their minimum. From every interior state, at most $m$ consecutive right steps reach $U$, with probability at least $p_{\ast }^m$. The Markov property then gives $\mathbb{P}(T>km)\le (1-p_{\ast }^m{)}^k$ for the first boundary hitting time. Thus $T$ is almost surely finite and has finite expectation, regardless of the holding probability of a Markov chain at each interior state.

// Target: probability-theory.bigb

The interior equation for a harmonic function for a Markov chain on the integers gives $p_k(h(k+1)-h(k))=q_k(h(k)-h(k-1))$. Positive scale increments satisfy $w_{k+1}=(q_k/p_k)w_k$. For absorbing boundaries $L<U$, the lower exit probability from $x$ isThis satisfies the harmonic equation and the boundary values and therefore is the hitting probability. For parity-dependent probabilities, $w_{j+2}=\rho w_j$ with $\rho =q_0{q}_1/(p_0{p}_1)$; geometric sums distinguish rightward, balanced and leftward escape.

// Target: markov-process.bigb

For a birth-death chain on the integers with right/left probabilities $p_i,q_i>0$ depending on parity $i\in \{0,1\}$, define $h_{\theta }(k)=k-\theta 1_{\{k\text{ odd}\}}$. It is a harmonic function for a Markov chain exactly whenequivalently $p_0{p}_1=q_0{q}_1$. The correction is bounded. Stopping this martingale at boundaries $-aN,bN$ gives a lower exit probability tending to $b/(a+b)$ as $N\to \infty$, even though the original position need not be a martingale.

// Target: markov-process.bigb

For a nearest-neighbour chain on the nonnegative integers, reflect at zero with holding probability $q_0$ and upward probability $p_0$. In a recurrent excursion from zero until its first return, expected visits $w_i$ satisfy $w_0=1$ and $w_i{p}_i=w_{i+1}{q}_{i+1}$, because each upward crossing is paired with a downward crossing. Thus the expected excursion length is ${\sum }_i{w}_i$ by Tonelli's theorem. Positive recurrence is equivalent to finiteness of that sum. The hitting-probability scale weights are ${\rho }_n={\prod }_{r=1}^n{q}_r/p_r$, and $w_n{\rho }_n=p_0/p_n$ helps show that summable excursion weights imply recurrence.