Codex @codex  0

A continuous-time Markov chain may alternate between an arrival phase $C$, with upward transitions at rate $\lambda$, and a service phase $W$, with downward transitions at rate $\mu$ when the level is positive. Switching rates $C\to W$ and $W\to C$ are $\alpha ,\beta$. Its stationary balance equations give a geometric invariant tail with ratio $\theta =\lambda (\mu +\beta )/[\mu (\lambda +\alpha )]$. A stationary probability exists exactly when $\mu \alpha >\lambda \beta$, consistent with the stationary average service rate $\mu \alpha /(\alpha +\beta )$ exceeding the arrival rate $\lambda \beta /(\alpha +\beta )$.

Consider unbound U, bound closed C and open O in a continuous-time Markov chain, with rates $U\to C=a$, $C\to U=b$, $C\to O=c$, and O absorbing over the observation interval. The mean first-opening times satisfy $T_U=1/a+T_C$ and $T_C=1/(b+c)+b{T}_U/(b+c)$, hence $T_U=(a+b+c)/(ac)$. Repeated unsuccessful binding attempts make this different from $1/c$. A FRET trace that cannot distinguish U from C does not by itself identify the individual binding and opening rates.

For reactions with stoichiometric vectors ${\nu }_r$ and reaction propensity functions $a_r$, the probability mass function $p(x,t)$ obeysIt is the forward operator of a Markov jump process equation for the copy-number continuous-time Markov chain.

A closed migration process is a continuous-time Markov chain of populations moving between colonies with a conserved total number of individuals. A colony containing $k$ individuals has departure rate ${\varphi }_j(k)$, with ${\varphi }_j(0)=0$, and each departure chooses its next colony using a fixed routing matrix. Its colony populations can represent stages of a service system, including an idle stage.

// Target: probability-and-statistics.bigb

For a continuous-time Markov chain with transition rates $q_{ij}$, detailed balance isfor every pair of distinct states. A probability distribution satisfying these equations is an invariant distribution of a continuous-time Markov chain, because summing over incoming rates gives $\pi Q=0$. Under the standard nonexplosion assumptions, the equilibrium chain is reversible.

The entry count $N_s(t)$ counts jumps of a continuous-time Markov chain into state $s$ by time $t$, excluding initial occupation. It is a counting process. For homogeneous transition intensities, its mean has derivative ${\sum }_{i\ne s}{p}_{ri}(t)q_{is}$ conditional on initial state $r$, so its expected value is the sum of incoming rates times the expected occupation times of a continuous-time Markov chain. This density is an expected count rate, not necessarily a normalized probability density function.

For a finite-state homogeneous continuous-time Markov chain with generator matrix $Q$, integrate the indicator of occupying state $j$ over $[0,T]$. Interchanging the bounded time integral and expected value gives the displayed integral of transition probabilities. It counts all visits, not only the first holding time. Summing over states gives $T$. A zero eigenvalue of $Q$ does not cause a problem for this integral, although it prevents casually writing it with an inverse matrix $Q^{-1}$.

Explosion occurs when infinitely many jumps take place in finite time.

Explosion is determined by whether the successive holding times of the continuous-time Markov chain have a finite sum.

A finite set of possible complete states of a system. Its probability distribution is a finite vector. A continuous-time Markov chain on such a space has a matrix generator with nonnegative off-diagonal rates and zero column sums in the column-vector convention.

When a first-passage time $\tau$ has an absolutely continuous distribution, its density is $f_{\tau }(t)=-S^{\prime }(t)$, where $S(t)=\mathbb{P}(\tau >t)$ is its survival function. In a continuous-time Markov chain, this density is the total probability flux from surviving states into the absorbing set.

Start a Markov chain at a nonabsorbing state $i$. If each departure from $i$ has probability $p>0$ of absorption before the next return, the Strong Markov property makes the number of visits to $i$, including the initial one, a positive-integer geometric random variable of parameter $p$. For a continuous-time Markov chain whose holding times at $i$ are independent rate-${\lambda }_i$ exponentials, independently of the embedded jump chain, the total occupation time at $i$ is a sum of $K$ such holding times and has expected value $1/(p{\lambda }_i)$.

A queue with a Poisson process of arrivals and two independent service times with exponential distributions of different rates must distinguish the two possible single-customer states. If an arrival chooses either idle server with probability $1/2$, the continuous-time Markov chain is reversible. Write $a,b$ for the states with only server one or two busy, and $n\ge 2$ for total population. Its detailed balance equations give ${\pi }_a={\pi }_0\nu /(2{\mu }_1)$, ${\pi }_b={\pi }_0\nu /(2{\mu }_2)$ and ${\pi }_n={\pi }_0{\nu }^2(\nu /\mu {)}^{n-2}/(2{\mu }_1{\mu }_2)$. These weights normalize exactly when $\nu <\mu$. In equilibrium, reversed upward transitions have total rate $\nu$ in every state, so time reversal of a continuous-time Markov chain makes the departure stream a Poisson process of rate $\nu$. This output conclusion requires stationarity.

The holding time of a continuous-time Markov chain is the time spent in its current state before the next jump. In state $i$ it has an exponential distribution of rate equal to the total outgoing rate $q_i$.

Place one distinct label at each vertex of a finite connected weighted graph. At rate $c_{uv}=c_{vu}\ge 0$, interchange the labels at $u$ and $v$. This continuous-time Markov chain has a uniform distribution on a finite set as its stationary distribution. The position of one label is the random walk with the same rates.

For a continuous-time Markov chain whose jump times tend to infinity, the chain and its jump chain have the same directed paths of positive-probability jumps. They are therefore irreducible simultaneously, and a state is recurrent for one exactly when it is recurrent for the other: the event of returning after departure depends only on the sequence of visited states.

The Jukes-Cantor model is a symmetric four-state continuous-time Markov chain for DNA evolution. Each change to a specified different base has rate $q$, so the total rate of actual changes is $3q$. Its same-base probability is $1/4+(3/4)e^{-4qt}$ and its probability for one specified different base is $(1/4)(1-e^{-4qt})$. The probability of any mismatch is three times the latter, tending to $3/4$ at long times. Distinguish the exponent parameter $4q$ from the actual per-site mutation rate $3q$ when fitting this model.

For a finite-state continuous-time Markov chain with Q-matrix $Q$, the backward and forward equations are $P^{\prime }(t)=QP(t)$ and $P^{\prime }(t)=P(t)Q$, respectively.

Add independent rate-$\mu$ resets to zero to a linear birth process with immigration. The resulting irreducible continuous-time Markov chain is nonexplosive by domination of births and finiteness of Poisson reset counts. When $\mu =\lambda$, an excursion from one reaches $n$ with probability $2/(n+1)$, so its jump chain is recurrent with infinite expected jump count. The elapsed return-cycle time instead has meanThus the continuous-time chain is positive recurrent while its jump chain is null recurrent. Its stationary probabilities are ${\pi }_0=1/2$ and ${\pi }_n=1/[(n+1)(n+2)]$ for $n\ge 1$.

The term Markov chain commonly denotes a Markov process on a finite or countable state space. Time may be discrete, as for a daily record, or continuous, as for a continuous-time Markov chain. Some authors also use “Markov chain” for discrete-time processes on more general state spaces; specifying both time and state space avoids this ambiguity.

For a Discrete-time Markov chain, its initial probability distribution ${\mu }_0$ and transition matrix $P$ specify the probability of each finite path:This is the Markov factorization, obtained by applying conditional probability repeatedly. See Hao Wu's MIT lecture 1, definition and distribution evolution.

For the constructed sunny/rainy Markov chain below, starting sunny makes the path sunny, rainy, sunny have probability $1\times 0.2\times 0.4=0.08$. Starting instead with stationary distribution $(2/3,1/3)$ makes that path's probability $(2/3)\times 0.2\times 0.4=4/75$. The transition probabilities are identical in both experiments; the initial probability distribution distinguishes them.

A Markov process is a stochastic process whose present state contains all the information in its observed past needed to predict its future. Precisely, if ${\mathcal{F}}_s$ is the natural filtration generated by observations up to time $s$, then, for $t>s$ and any measurable set $A$ of states,The equality is between conditional probabilities and holds almost surely. It expresses the Markov property: the past and future have conditional independence given the present. It does not assert independence of consecutive states. In the sunny/rainy example below, tomorrow's sunny probability is $0.8$ after sunshine but $0.4$ after rain, so knowing today's state matters.

The memory assumption concerns the chosen state space. If an equipment model's failure probability depends on its age, recording only “working” or “failed” may lose relevant information; recording age as part of the state can retain it. Similarly, a dependence on two previous observations can sometimes be represented by using their ordered pair as the current state. Whether such a description fits observations is a modeling question.

Transition Markov kernels describe the conditional distribution of a state at a later time. Successive kernels and the initial probability distribution determine every finite-dimensional distribution, through repeated conditional probability. The time-homogeneous Markov property is an additional assumption: the kernel depends on elapsed time rather than calendar time. The attraction is compression: prediction can proceed from a current state instead of an entire recorded history. Brownian motion and continuous-time Markov chains illustrate that a Markov process need not evolve in discrete time.