Continuous-time Markov chain

A continuous-time Markov chain waits an exponential time of rate $q_i$ in state $i$ and then jumps according to probabilities $p_{ij}=q_{ij}/q_i$.

A continuous-time Markov chain is nonexplosive if it makes only finitely many jumps on every bounded time interval. A uniformly bounded total jump intensity permits domination by a Poisson process. More generally, if a population process stays in a finite random set of states on each finite interval and every individual state has finite exit rate, localizing to fixed finite sets gives the same conclusion. In an open migration process, external Poisson immigration bounds the total population while internal moves preserve it and exits reduce it, giving this localization.

A Markov jump process is a nonexplosive, piecewise-constant continuous-time Markov process. Transition rates $q(x,y)$ give exponential holding time of rate ${\sum }_{y\ne x}q(x,y)$ and next-state probabilities proportional to these rates. Being Markov relative to a specified filtration requires the conditional future law given that filtration to depend only on the present state.

// Target: probability-and-statistics.bigb

A Markov chain $Z$ with transition kernel $K$, sampled at an independent Poisson process $N$ of rate $r$, is a pseudo-Poisson process. Its infinitesimal generator is $r(K-I)$ and its transition semigroup is $e^{-rt}{\sum }_{n\ge 0}(rt{)}^n{K}^n/n!$. Conditioning on $N_t$ proves the latter formula, and differentiation at zero gives the former. Its resolvent operator is $(\lambda +r{)}^{-1}{\sum }_{n\ge 0}[r/(\lambda +r){]}^n{K}^n$ for $\lambda >0$, by integrating the nonnegative scalar Poisson weights. The series converge in operator norm on bounded Borel measurable functions.

This finite-state Markov jump process models two opposing populations, each suffering losses at a rate proportional to the surviving opposing population. Both coordinate axes are absorbing. Its embedded jump chain removes one individual at each step, with probabilities $b/(a+b)$ and $a/(a+b)$. The corresponding deterministic equations $\dot{a}=-\kappa b$, $\dot{b}=-\kappa a$ conserve $a^2-b^2$; finite stochastic populations satisfy a near-invariant instead.

In the jump chain of the symmetric stochastic attrition process, the two possible increments of $D$ are $-2A+1$ and $2B-1$. Their mean is the displayed quantity, bounded by one in absolute value. Subtracting these conditional means produces a martingale. If both initial populations are at most $N$, it has at most $2N$ nonzero increments, each bounded by $4N$. Thus Azuma-Hoeffding inequality controls changes of order $N^2$ with exponentially small probability of order $e^{-cN}$.

Start the symmetric stochastic attrition process at $(N,\lfloor \alpha N\rfloor )$, with $0<\alpha <1$, and let $X^N$ be the final first-population size divided by $N$. The bound holds for all sufficiently large $N$, depending on $\alpha ,\delta >0$. At absorption, one coordinate is zero. If the first survives, the terminal squared difference is $N^2(X^N{)}^2$; if it is extinguished, that difference is nonpositive. In either case the indicated deviation forces an error of at least $\sqrt{1-{\alpha }^2}\delta N^2$ from the limiting squared invariant. The initial rounding error and total conditional drift are only $O(N)$. Apply Azuma-Hoeffding inequality to the centered squared-population near-invariant for stochastic attrition, using its $2N$ increments bounded by $4N$, to obtain the displayed exponent.

Let $T$ be a rate-one exponential random variable independent of a uniform random variable $U$ on $(0,1)$. In its natural filtration, this Markov jump process jumps once from zero to its mark and then stays there. Its destination rate kernel is $q(x,dy)=1_{\{x=0\}}{1}_{(0,1)}(y)dy$. For a Borel mark set $B$, the martingale $1_{\{T\le t,U\in B\}}-|B|(t\wedge T)$ follows directly from the memoryless residual lifetime. Information about the future mark or lifetime must not be inserted into the initial filtration.

A kernel specifying the jump rate from state $x$ for each displacement $z$. With the displacement convention the infinitesimal generator is $L\phi (x)=\int (\phi (x+z)-\phi (x))K(x,dz)$. The destination convention instead records a rate measure on the new state, translating each displacement by $x$. For finite total rate, that total is the exponential holding rate and the normalized measure is the next-jump distribution.

With the displacement convention for a Lévy kernel of a Markov jump process, suppose its total rate is bounded, its first moment is zero and its second moment $v$ is bounded by $V$. The bounded rate prevents explosion. The compensated displacement integral is a square-integrable martingale with $\mathbb{E}{X}_t^2\le Vt$. The displayed process is the sum of $2\int X_{s-}d{X}_s$ and the compensated sum of squared jumps. The first term is square integrable since $4\mathbb{E}{\int }_0^t{X}_{s-}^{2}v(X_{s-})ds\le 2V^2{t}^2$; the second is integrable because its expected total jump-square sum is at most $Vt$. Consequently the square compensation is a true martingale, without a fourth-moment assumption.

For a Markov jump process with off-diagonal rate kernel $q(x,dy)$, the jump measure records the jump time and the destination state. Relative to a filtration for which the process has the Markov jump property, its predictable compensator is the displayed rate measure, up to explosion. The corresponding generator is $Qf(x)=\int [f(y)-f(x)]q(x,dy)$. An enlarged filtration revealing future jumps need not preserve this compensator, so the filtration condition is essential.

The predictable random measure $\nu$ matching the expected integrals of the jump measure $N$ against every nonnegative predictable integrand. For displacement marks of a Markov jump process, $\nu (ds,dy)=K(X_{s-},dy)ds$. Multiplying the integrand by an event known before an interval proves that the compensated integral has conditionally centered increments; expected absolute integrability makes it a true martingale. Squared-integrand integrability gives the square-integrable compensation isometry.

A predictable mark-dependent integrand $H(t,y)$ gives a true martingale ${\int }_0^t\int H(s,y)(\mu -\nu )(ds,dy)$ whenever $\mathbb{E}{\int }_0^t\int |H|d\nu <\infty$ on every finite horizon. Local versions follow when this expected integrability holds after localization. The compensator identity applied to predictable indicators from the past proves that every increment has conditional expectation zero. Choosing $H(s,y)=f(y)-f(X_{s-})$ gives the generator martingale $f(X_t)-f(X_0)-{\int }_0^{t}Qf(X_s)ds$ under the same local integrability conditions.

With scale $N$, such a process has finitely many possible jump vectors $\ell /N$ occurring at rates $N{\beta }_{\ell }(x)$ depending on the current scaled state. Its drift is ${\sum }_{\ell }\ell {\beta }_{\ell }(x)$, while the predictable covariance rate of its martingale fluctuations is $N^{-1}{\sum }_{\ell }\ell {\ell }^T{\beta }_{\ell }(x)$.

// Target: probability-and-statistics.bigb

In a population of $N$ playing rock paper scissors, let losers copy winners. With pair-selection events of rate $(N-1)/2$, the proportion process has jumps $(e_1-e_3)/N$, $(e_2-e_1)/N$, $(e_3-e_2)/N$ at rates $N{x}_1{x}_3$, $N{x}_2{x}_1$, $N{x}_3{x}_2$. Its fluid limit obeys ${\dot{x}}_i=x_i(x_{i-1}-x_{i+1})$, with indices interpreted cyclically. The product of numbers $x_1{x}_2{x}_3$ is conserved in this ordinary differential equation, whereas the finite chain eventually reaches a monochromatic absorbing state.

For the cyclic imitation chain, its infinitesimal generator satisfies ${\mathcal{L}}_N(x_1{x}_2{x}_3)=-3x_1{x}_2{x}_3/N$. Therefore $\mathbb{E}[X_1^N(t)X_2^N(t)X_3^N(t)]=e^{-3t/N}\mathbb{E}[X_1^N(0)X_2^N(0)X_3^N(0)]$. While all three types remain, the product of numbers is at least $(N-2)/N^3$ for $N\ge 3$, so this identity proves that one type is lost almost surely. The remaining pair has a fixed winner, which eventually occupies the whole population. The conserved product in the deterministic fluid limit therefore describes fixed finite horizons, not the chain's infinite-time behavior.

With jumps $\ell /N$ at rates $N{\beta }_{\ell }(z)$, compensate each jump count by its integrated rate. The drift is $F(z)={\sum }_{\ell }\ell {\beta }_{\ell }(z)$ and the predictable quadratic variation matrix of the martingale term is $N^{-1}{\int }_0^t{\sum }_{\ell }\ell {\ell }^{\mathsf{T}}{\beta }_{\ell }(Z_s^N)ds$. On a compact region with bounded rates this bracket is of order $1/N$. A Lipschitz drift and the Gronwall inequality transfer martingale concentration to the fluid limit.

For jump rates $q_{ij}=r_j$ when $i\ne j$, let $R={\sum }_j{r}_j>0$. The positive-rate states form the unique closed communicating class; every zero-rate state is an open singleton. The stationary law is ${\pi }_i=r_i/R$, with detailed balance ${\pi }_i{q}_{ij}=r_i{r}_j/R$. When at least two rates are positive, the embedded jump-chain law is proportional to $r_i(R-r_i)$. With only one positive rate, that state is absorbing and the genuine sequence of jump times stops there; an extended discrete chain can use an absorbing self-loop.

On nonnegative levels, a chain moves right at rate $\lambda$ in state $C$, left at rate $\mu$ in state $W$, and switches $C\to W$ at rate $\alpha$ and $W\to C$ at rate $\beta$. The minimal nonnegative solution for hitting $W_0$ is identically one when $\mu \alpha \ge \lambda \beta$. When $\mu \alpha <\lambda \beta$, it is $h_{W_i}={\vartheta }^i$ and $h_{C_i}=b{\vartheta }^i$, where $b=\alpha (\mu +\beta )/[\beta (\lambda +\alpha )]<1$. These cases give recurrence and transience respectively.

A continuous-time Markov chain with finite holding rates is nonexplosive when its jump times have no finite accumulation almost surely. It then remains in its specified state space at every finite time, rather than being lost to a cemetery state after infinitely many jumps. Bounded rates are sufficient for nonexplosion, but finite rates at individual states alone are not.

For a nonexplosive continuous-time Markov chain on the nonnegative integers, suppose zero is absorbing and each positive state has positive rate to zero. On every finite positive set the conditional chance of absorption at the next jump has a positive minimum. By the Strong Markov property, the chance of surviving $r$ departures from that set is bounded by $(1-\delta {)}^r$. Without absorption only finitely many such departures occur, and finite positive holding rates prohibit staying forever at a positive state. Thus eventually the path stays outside each finite set: it escapes to infinity. This conclusion needs no upward skip-free assumption.

An equilibrium probability row vector is invariant under every time transition matrix, and satisfies $\pi Q=0$ when generator sums are defined. A finite irreducible chain has a unique such distribution and all transition rows converge to it. With absorbing states, every probability mixture on the closed absorbing states is an equilibrium, and limiting rows are determined by absorption probabilities. For a jump chain with stationary weights ${\alpha }_i$ and positive holding rates ${\lambda }_i$, the corresponding time-stationary weights are proportional to ${\alpha }_i/{\lambda }_i$.

For a finite-state continuous-time Markov chain, a transition intensity matrix has nonnegative off-diagonal rates $q_{rs}$ and $q_{rr}=-{\sum }_{s\ne r}{q}_{rs}$. Every row sums to zero. In the time-homogeneous case its transition probability matrix is the matrix exponential $P(t)=e^{tQ}$. An absorbing state has a zero row.

For a finite irreducible Markov chain in continuous time, time averages converge almost surely to averages under its unique stationary distribution, from any initial state. Applying this to an activation indicator yields the corresponding throughput.

A continuous-time Markov chain is nonexplosive when it makes only finitely many jumps in every finite time interval, almost surely. Uniformly bounded total jump rates suffice. Another useful criterion bounds the states visited over each finite interval by a random finite set on which rates are bounded.

Global balance equates total stationary incoming and outgoing transition intensities at each state. Detailed balance for a continuous-time Markov chain is sufficient but stronger: it equates each pair separately.

A jump process with event types $k$, increments ${\nu }_k$, and rates $a_k(X)$ can be represented as $X(t)=X(0)+{\sum }_k{\nu }_k{P}_k({\int }_0^t{a}_k(X(s-))ds)$, with independent unit-rate Poisson processes. This represents the Markov jump-process generator through time-changed counts.

Place one distinct label at each vertex of a finite connected weighted graph. At rate $c_{uv}=c_{vu}\ge 0$, interchange the labels at $u$ and $v$. This continuous-time Markov chain has a uniform distribution on a finite set as its stationary distribution. The position of one label is the random walk with the same rates.

On a finite connected weighted graph, the interchange process and the single-label random walk with the same symmetric rates have equal spectral gaps. This is a substantial theorem, not just the elementary inclusion of the single-label spectrum in the full spectrum. A proof is given in Proof of Aldous' spectral gap conjecture, Theorem 1.1.

For an equilibrium chain with strictly positive stationary probabilities, reversed jump rates are $q_{ij}^{\ast }={\pi }_j{q}_{ji}/{\pi }_i$. Under detailed balance they equal the original rates, making the stationary path law reversible. Holding times and jump directions reverse together; this is more than symmetry of one-time distributions.

A partition-valued Markov chain in which every unordered pair of current blocks merges at rate one. On $n$ initial singleton blocks, the holding time with $j$ blocks is exponential with rate $\left(\genfrac{}{}{0ex}{}{j}{2}\right)$. Restrictions to subsets of labels have the same law.

In Kingman's coalescent, the epoch with $j$ ancestral lineages lasts $T_j\sim \mathrm{Exp}((\genfrac{}{}{0ex}{}{j}{2}))$, independently across epochs. If each lineage carries an independent Poisson process of mutations with rate $\theta /2$, its epoch mutation count satisfies $Y_j\mid T_j\sim \mathrm{Poisson}(j\theta T_j/2)$. The total mutation count is $S={\sum }_{j=2}^n{Y}_j$. Under the infinite sites mutation model, this also counts segregating sites; without that assumption it still counts mutation events, which need not correspond one-to-one to observed polymorphisms.

The independent mutation counts in coalescent epochs have the geometric mutation count in a coalescent epoch laws. Summing their expectations and variances gives the displayed formulas, where $H_m={\sum }_{i=1}^{m}1/i$ and $H_m^{(2)}={\sum }_{i=1}^{m}1/i^2$. The extra quadratic term records randomness of the ancestral tree, beyond the conditional Poisson distribution variance.

Integrating the Poisson distribution probability generating function over the exponential distribution of the epoch length givesThus $Y_j$ has a geometric distribution on $\{0,1,\dots \}$, with expectation $\theta /(j-1)$ and variance $\theta /(j-1)+{\theta }^2/(j-1{)}^2$. The support convention matters: the count includes zero mutations.

The epoch lengths in Kingman's coalescent are independent random variables. Conditional on these lengths, mutation counts on disjoint tree pieces are independent Poisson random variables. Their conditional laws depend only on the corresponding epoch length, so integrating the product of conditional laws against the independent epoch laws leaves independent unconditional epoch mutation counts. This justifies adding their variances, rather than assuming independence from conditional independence alone.

In coalescent units, independent waiting times satisfy $T_j\sim \mathrm{Exp}(j(j-1)/2)$, and total branch length is $L={\sum }_{j=2}^{n}j{T}_j$. Therefore $EL=2{\sum }_{i=1}^{n-1}1/i$. The variables $j{T}_j/2$ are independent exponentials with rates $j-1$, so $W=L/2$ has the same distribution as the maximum of $n-1$ independent unit exponentials, with density $(n-1)e^{-w}(1-e^{-w}{)}^{n-2}$. The last identity follows either from its Laplace transform or from the independent spacings of exponential order statistics.

The consistent partition process on $\mathbb{N}$ whose finite restrictions are Kingman's coalescent. It starts with infinitely many singleton blocks.

For $t>0$, the number $B_t$ of blocks is finite almost surely. On a finite restriction, the expected time to reach $m$ blocks is at most ${\sum }_{j>m}2/[j(j-1)]=2/m$. Markov inequality, followed by increasing the restriction, gives $\mathbb{P}(B_t>m)\le 2/(mt)$. Let $m\to \infty$, then use rational times and monotonicity to obtain the assertion simultaneously for every $t>0$.

The entry count $N_s(t)$ counts jumps of a continuous-time Markov chain into state $s$ by time $t$, excluding initial occupation. It is a counting process. For homogeneous transition intensities, its mean has derivative ${\sum }_{i\ne s}{p}_{ri}(t)q_{is}$ conditional on initial state $r$, so its expected value is the sum of incoming rates times the expected occupation times of a continuous-time Markov chain. This density is an expected count rate, not necessarily a normalized probability density function.

The occupation time records the duration spent in a state of a continuous-time Markov chain. Conditional on $Z(0)=r$, its expected value is ${\int }_0^t{p}_{rs}(u)du$, where $p_{rs}$ is the transition probability.

For a continuous-time Markov chain with absorbing endpoints $0,N$ and rates $q_{j,j\pm 1}={\lambda }_j/2$, the jump chain is a simple symmetric random walk. On leaving $i$, absorption before returning has probability $p_i=\frac{1}{2}(1/i+1/(N-i))$, by gambler's ruin on each side. The geometric return count before absorption and independent rate-${\lambda }_i$ holding times give ${\mathbb{E}}_{i}I=1/(p_i{\lambda }_i)$. The rates at other states affect elapsed excursion times but not their return probabilities.

For a finite-state homogeneous continuous-time Markov chain with generator matrix $Q$, integrate the indicator of occupying state $j$ over $[0,T]$. Interchanging the bounded time integral and expected value gives the displayed integral of transition probabilities. It counts all visits, not only the first holding time. Summing over states gives $T$. A zero eigenvalue of $Q$ does not cause a problem for this integral, although it prevents casually writing it with an inverse matrix $Q^{-1}$.

The holding time of a continuous-time Markov chain is the time spent in its current state before the next jump. In state $i$ it has an exponential distribution of rate equal to the total outgoing rate $q_i$.

A state's holding time in a time-homogeneous continuous-time Markov chain is exponentially distributed with rate $-q_{rr}>0$. Its mean is $-1/q_{rr}$; the probability that its next jump is to state $s$ is $q_{rs}/(-q_{rr})$. An absorbing state has infinite holding time.

For an exponential distribution with positive rate $\lambda$, the mean is $1/\lambda$. A confidence interval for the rate transforms to one for the mean by reversing and inverting its endpoints. The monotonic transformation preserves the coverage event; it does not give a prediction interval for one realized holding time.

For transition rate $w_{ij}$ from state $i$ to state $j$, the master equation is

A directed transition count is its conditional jump rate multiplied by the occupation probability of its departure state. Net current is the forward count minus the reverse count, multiplied by the transported charge. On a stationary finite tree each edge has zero net current, although both directed counts can be nonzero. Different edges can have different traffic.

For edge current $J_{ij}=w_{ij}{P}_i-w_{ji}{P}_j$, the total entropy production rate isIt vanishes precisely when every occupied edge satisfies detailed balance.

A continuous-time random walk waits a random time at each site before jumping. Constant outgoing rates produce an exponential residence time, while the relative transition rates determine the destination of the next jump.

Particles perform independent continuous-time random walks until they meet, after which they share one path. A finite initial particle set remains finite and its cardinality cannot increase. They arise as the common ancestors of sites in a voter model. On the square lattice every pair eventually coalesces, because the difference walk is recurrent Markov chain.

A finite-state process has generator $Q$ exactly when, for every function $f$ on the state space,is a martingale with respect to the process filtration.

A Q-matrix has nonnegative off-diagonal entries and row sums zero. Its diagonal entry is the negative total rate out of the state.

For a time-homogeneous continuous-time Markov chain, $P_{ij}(t)={\mathbb{P}}_i(X_t=j)$ and the matrices satisfy $P(s+t)=P(s)P(t)$, the Chapman-Kolmogorov equation. On a finite state space with Q-matrix $Q$, the matrix exponential gives $P(t)=e^{tQ}$.

The distinction from a discrete-time transition matrix is the elapsed-time parameter: $P(t)$ describes where the process is after $t$ time units, allowing any number of intervening jumps. Each $P(t)$ is a stochastic matrix, with $P(0)=I$. A Q-matrix instead records instantaneous rates and has row sums zero. Sampling at a fixed spacing $\Delta$ yields a Discrete-time Markov chain with transition matrix $P(\Delta )$. See Hao Wu's MIT lecture 21, transition semigroups and lecture 22, infinitesimal rates.

Sample a finite-state continuous-time Markov chain with generator $Q$ at the independent arrival times of a rate-$\lambda$ Poisson process. The sampled states form a discrete-time Markov chain with transition matrixIts invariant distributions are exactly those of the continuous-time chain.

For a finite-state continuous-time Markov chain with Q-matrix $Q$, the backward and forward equations are $P^{\prime }(t)=QP(t)$ and $P^{\prime }(t)=P(t)Q$, respectively.

A Markov reward model attaches a reward rate to each state and integrates it over the time spent there.

For transition rates $q_{nm}$, the generator acts on a function $f$ byWhenever the expectations are finite,Choosing polynomial functions $f$ gives differential equations for the moments.

For a positive function $g$ in the domain of a Markov jump-process generator, the generator martingale for $g(X_t)$ and semimartingale integration by parts give the displayed local martingale. Suitable finite-horizon deterministic bounds on $g$ and $Lg/g$ make it a true martingale. For a single marked exponential jump process, take $g(x)=e^{\theta x}$. Then the exponent's compensating process is $r_{\theta }(t\wedge T)$, where $r_{\theta }=(e^{\theta }-1)/\theta -1$ for $\theta \ne 0$ and $r_0=0$; it is continuous adapted and hence predictable.

For jumps $x\mapsto x+{\nu }_r$ of rates $a_r(x)$, the forward operator acts on a probability mass function byIt is adjoint to the Markov jump-process generator $Lf(x)={\sum }_r{a}_r(x)[f(x+{\nu }_r)-f(x)]$.

Conditioned on a finite jump path $i_0,\dots ,i_n$, the holding times are independent exponentials with the corresponding rates. Multiplying the probability of occupying $i_n$ at time $t$ by $q_{i_n}$ turns its simplex density into a product symmetric under reversal of the time portions and the state sequence.

For $h>0$, the fixed-time skeleton has matrix $P(h)$. If $q_i$ is the holding rate, then for $nh\le t\le (n+1)h$,Thus ${\int }_0^{\infty }{p}_{ii}(t)dt$ diverges exactly when ${\sum }_n{p}_{ii}(nh)$ does, proving recurrence equivalence for an irreducible chain.

The jump chain records the successive states visited by a continuous-time Markov chain while discarding its holding times.

On $\mathbb{N}$, let the chain jump from every $i\ge 1$ to $i+1$ with probability $p$ and to $i-1$ with probability $1-p$, and jump from zero to one. It is recurrent exactly when $p\le 1/2$; at $p=1/2$ it is null recurrent.

For holding rates $q_i$, a measure $\pi$ satisfies $\pi Q=0$ exactly when ${\mu }_i=q_i{\pi }_i$ is invariant for the jump chain. Thus ${\pi }_i={\mu }_i/q_i$ when normalization is possible.

Even when the jump chain has invariant probability $\mu$, the continuous-time invariant measure ${\mu }_i/q_i$ may have infinite mass if holding rates become too small, producing infinite mean return time.

A recurrent jump chain with infinite invariant measure $\mu$ can yield a positive-recurrent continuous-time chain when ${\sum }_i{\mu }_i/q_i<\infty$.

If the jump chain visits a state of finite holding rate infinitely often, the sum of its independent positive holding times at that state diverges almost surely, preventing explosion.

For a continuous-time Markov chain whose jump times tend to infinity, the chain and its jump chain have the same directed paths of positive-probability jumps. They are therefore irreducible simultaneously, and a state is recurrent for one exactly when it is recurrent for the other: the event of returning after departure depends only on the sequence of visited states.

Explosion occurs when infinitely many jumps take place in finite time.

Explosion is determined by whether the successive holding times of the continuous-time Markov chain have a finite sum.

If every holding rate of a continuous-time Markov chain is at most $C<\infty$, its holding times can be coupled as $E_n/q_{Y_n}\ge E_n/C$, where the $E_n$ are independent rate-one exponential random variables. The strong law of large numbers gives ${\sum }_n{E}_n=\infty$ almost surely, so the jump times tend to infinity and the chain cannot explode.

Conditional on a jump path $(Y_n)$, if ${\sum }_{n}1/q_{Y_n}<\infty$, then the sum of exponential holding times has finite expectation and is finite almost surely.

Suppose a transient nearest-neighbour jump chain on $\mathbb{N}$ has uniformly bounded expected visits to each state and the holding rate at $i$ grows geometrically. Thenso the continuous-time chain explodes almost surely.

For a finite-state chain with transition semigroup $P(t)=e^{tQ}$, a probability row vector $\pi$ is invariant when $\pi P(t)=\pi$ for every $t\ge 0$. Equivalently,

A countable nonexplosive continuous-time Markov chain which is an irreducible Markov chain and has an invariant distribution $\pi$ is positive recurrent; its transition probabilities tend to $\pi$ from every initial Markov chain state.

To see why continuous time removes periodicity, fix $h>0$. Every finite sequence of Markov chain states connected by positive transition intensities has positive probability of finishing before $h$, followed by no further transition until $h$. Consequently the transition matrix $P_h$ has positive entries between every pair of communicating Markov chain states, including its diagonal. It is an irreducible Markov chain and an aperiodic Markov chain with stationary distribution $\pi$. The countable-state Markov chain convergence theorem gives convergence at times $nh$, also in total variation distance. For $t=nh+r$, $0\le r<h$, applying $P_r$ contracts total variation distance and preserves $\pi$, giving the full continuous-time limit.

An invariant distribution of a continuous-time Markov chain whose coordinates are independent Poisson distributions. This property needs verification from the reaction propensity functions; it does not follow merely from having an equilibrium for the deterministic reaction-rate equations.

For example, consider the cyclic chemical reaction network with stoichiometric vectors $(1,0)$, $(0,1)$ and $(-1,-1)$ and respective reaction propensity functions $\lambda$, $\beta n$, $\gamma nm$, where all three constants are positive. Put $a=\lambda /\beta$ and $b=\beta /\gamma$. For $\pi (n,m)=e^{-a-b}{a}^n{b}^m/(n!m!)$, the three incoming terms in the chemical master equation are respectively $\beta n\pi (n,m)$, $\gamma nm\pi (n,m)$ and $\lambda \pi (n,m)$. They exactly cancel the outgoing terms. Boundary terms vanish by the zero negative-index convention, and normalization follows from the two Poisson distributions. Establishing a limiting probability distribution additionally requires nonexplosion, an irreducible Markov chain and positive recurrence.

Every closed communicating class of a finite Markov chain has its own invariant probability distribution. All invariant distributions are convex combinations of those class distributions. In particular, a finite irreducible chain has a unique invariant distribution, while a reducible chain may have many.

An explosive chain can have a summable solution of $\pi Q=0$ despite being transient. Nonexplosion is needed for the usual equivalence between invariant distributions and positive recurrence.

A batch-birth linear-death process has transitionsWriting $P_n(t)=\mathbb{P}(N_t=n)$ and taking $P_j=0$ for $j<0$, its master equation is

For $m=\mathbb{E}N$ and $v=\mathrm{var}N$, the Markov jump-process generator applied to $n$ and $n^2$ givesConsequently

A birth-death process moves only between neighbouring nonnegative integer states, with state-dependent birth and death rates.

A pure birth process on ${\mathbb{N}}_0$ jumps only from $n$ to $n+1$, at a strictly positive rate ${\lambda }_n$. Its successive holding times are independent exponential variables of rates ${\lambda }_n$.

For a pure birth process with fixed initial state and finite statewise rates, the jump-count compensator gives $dM=(e^{\theta }-1)M_{-}[dN-\lambda (X_{-})dt]$, so $M$ is a local martingale up to explosion. If $\lambda \le C$, a rate-$C$ Poisson candidate clock with accepted jumps of probability $\lambda (X_{-})/C$ prevents explosion and dominates the jump count. Its finite exponential moments dominate every stopped $M$ on each finite horizon, upgrading the local martingale to a true martingale. A random initial state requires $\mathbb{E}{e}^{\theta X_0}<\infty$; uniformly bounded rates do not supply this missing initial integrability.

For a carrying capacity $N$, each of $k$ cells splits into two at rate $1-k/N$, so the population increases by one at the displayed total rate and is absorbed at $N$. Starting from population density $p\in (0,1)$, its normalized density converges on finite horizons to $p{e}^t/(1-p+p{e}^t)$ by the Kurtz fluid limit theorem.

// Target: analysis.bigb

A pure birth process is nonexplosive exactly whenIts explosion time is the sum of independent exponential holding times. If the displayed series converges, that sum has finite expectation. If it diverges, the Laplace transform of the sum isforcing the sum to be infinite almost surely.

The pure birth process with rates ${\lambda }_n=\alpha n+\beta$, where $\alpha ,\beta >0$, combines linear reproduction with immigration. Starting from zero, its mean satisfiesand hence