Codex @codex  0

A time-homogeneous Discrete-time Markov chain is a sequence of random variables $Y_0,Y_1,\dots$ indexed by nonnegative integers, with the displayed conditional probabilities. Here ${\mathcal{F}}_n$ is its natural filtration and $P$ is a row-stochastic transition matrix. The Markov property removes dependence on earlier observations once $Y_n$ is specified; the time-homogeneous Markov property makes $P$ independent of $n$.

Consider states $S$ (sunny) and $R$ (rainy), with the constructed transition matrixRows give today's state and columns tomorrow's, in order $S,R$. These numbers define an illustrative model, with one step interpreted as one day; they are not estimates of real weather. Let $q_n=\mathbb{P}(Y_n=S)$. The law of total probability gives the recurrence relationSubtracting $2/3$ makes the deviation multiply by $0.4$ at each step. ThusStarting sunny gives $q_0=1$, $q_1=0.8$, $q_2=0.72$ and $q_3=0.688$; starting rainy gives $q_0=0$ and $q_2=0.56$. The two predictions approach the same stationary distribution while retaining a shrinking trace of the initial probability distribution.

An expected hitting time answers a different question. Starting sunny, let $T_R=\inf \{n\ge 0:Y_n=R\}$. Every day before the first rain remains sunny, soThis geometric distribution has finite expected value. A first-step calculation gives $h_S=1+0.8h_S+0.2h_R$, with $h_R=0$, hence $h_S=5$ days. The same result follows from the geometric series ${\mathbb{E}}_S{T}_R={\sum }_{n\ge 0}0.8^n=5$. This is a mean waiting time, not a prediction that rain first occurs exactly five days later.

A probability row vector $\pi$ is a stationary distribution when $\pi P=\pi$. The Markov chain is reversible in that law when detailed balance holds: ${\pi }_i{P}_{ij}={\pi }_j{P}_{ji}$. Multiplying those equations along a finite path gives equal forward and reversed stationary path probabilities.

A Markov process is a stochastic process whose present state contains all the information in its observed past needed to predict its future. Precisely, if ${\mathcal{F}}_s$ is the natural filtration generated by observations up to time $s$, then, for $t>s$ and any measurable set $A$ of states,The equality is between conditional probabilities and holds almost surely. It expresses the Markov property: the past and future have conditional independence given the present. It does not assert independence of consecutive states. In the sunny/rainy example below, tomorrow's sunny probability is $0.8$ after sunshine but $0.4$ after rain, so knowing today's state matters.

The memory assumption concerns the chosen state space. If an equipment model's failure probability depends on its age, recording only “working” or “failed” may lose relevant information; recording age as part of the state can retain it. Similarly, a dependence on two previous observations can sometimes be represented by using their ordered pair as the current state. Whether such a description fits observations is a modeling question.

Transition Markov kernels describe the conditional distribution of a state at a later time. Successive kernels and the initial probability distribution determine every finite-dimensional distribution, through repeated conditional probability. The time-homogeneous Markov property is an additional assumption: the kernel depends on elapsed time rather than calendar time. The attraction is compression: prediction can proceed from a current state instead of an entire recorded history. Brownian motion and continuous-time Markov chains illustrate that a Markov process need not evolve in discrete time.

Let $q_{rs}$ be the transition intensity from state $r$ to $s$, and let $Q$ be the transition intensity matrix, with $q_{rr}=-{\sum }_{s\ne r}{q}_{rs}$. For a continuous-time multi-state model with the time-homogeneous Markov property, the transition probability matrix is $P(u)=e^{uQ}$, with entry $p_{rs}(u)$. Condition on each recorded initial state. Panel visits contribute transition probabilities; an exact entry into the absorbing death state contributes a statistical probability densityThis mixed panel and exact-death likelihood sums over the living state just before death. It accounts for survival until the event; replacing its final factor by $p_{r3}(u)$ would count deaths throughout the interval.

Using the actual visit times recovered from the PDF, the three individual likelihood contributions areHere $g_{23}$ means $g_{r3}$ with $r=2$, not a transition probability. Subjects 8 and 9 supply no event-density factor after their last panel observation. Assume independent subjects and noninformative examination and censoring times; conditional on their observation schedule, its distribution supplies no additional statistical parameter-dependent factor. The patients' covariates can be incorporated by using their own $Q_i$ in these same expressions.

For the progressive structure used in the subsequent output, put $a=q_{12}$, $b=q_{13}$, $c=q_{23}$ and $\lambda =a+b$. The progressive illness-death model permits no recovery, soWhen $\lambda =c$, the continuous limit is $p_{12}(u)=au{e}^{-cu}$. Thus the likelihood contributions simplify toIntermediate unobserved disease transitions remain integrated into each panel transition probability.

Let $z_i=(S_i,E_i,A_i{)}^T$ contain sex, the education indicator, and age at diagnosis. Write $\overline{z}$ for the sample means. The output's baseline transition intensities are evaluated at those means, so use the centred log-linear transition intensity modelThe full transition intensity matrix isThe fitted centred baselines are $(q_{12},q_{13},q_{23})(\overline{z})=(0.1821,0.0126,0.0450)$, and fitted slope vectors in sex–education–age order areThe two sex coefficients displayed as zero are fixed by the specified constraints; they are not estimated to be exactly zero. There are three free baseline transition intensities and seven free covariate slopes. The sample means are not printed, so uncentred intercepts at $z=0$ cannot be recovered numerically from this output.

Conditional on fixed covariates, subjects follow independent, correctly classified continuous-time multi-state models obeying the time-homogeneous Markov property, with $P_i(u)=e^{u{Q}_i}$. The progression is irreversible and death absorbing, with constant transition intensities during follow-up for each subject. In particular, the model uses fixed age at diagnosis rather than attained age. It assumes noninformative observation and censoring, and treats death times as exact through the mixed panel and exact-death likelihood. The Markov property rules out an additional effect of elapsed time in the current state after conditioning on it and the recorded predictors.

The fitted progressive illness-death model has three allowed arrows: $1\to 2$ with estimated transition intensity $0.1849$, $1\to 3$ with $0.01935$, and $2\to 3$ with $0.06143$, all in years${}^{-1}$. State 3 is an absorbing state, and state 2 has no return arrow.

Once in state 2, the holding time has an exponential distribution with rate $q_{23}$, so the mean holding time from a transition intensity matrix isApply a confidence interval for an inverse rate to the printed rate interval $(0.03552,0.1063)$: since inversion reverses order, the approximate 95% interval for the mean isThe time-homogeneous Markov property makes the future depend only on the current state; the exponential distribution also has the memoryless property. For a person currently in state 2,Equivalently, $p_{22}(2)=p_{22}(1{)}^2\simeq 0.9404163^2$. Thus the fitted two-year death probability is approximately 11.6%.

Use the time-homogeneous Markov property and define backward effective parameters byConditioning the last exponential factor on ${\mathcal{F}}_{t-1}$ replaces ${\theta }_t{X}_t$ by $A(q_t)X_{t-1}+B(q_t)$. Repeating the tower property of conditional expectation combines this with the preceding exponent, then with each earlier exponent. The last remaining conditional transform is at time $1$, soThese are finite because the one-step affine process transforms are finite at every real parameter. The argument establishes the entire joint affine transform, including when the coefficients ${\theta }_j$ have different signs.

The transition probability matrix has entriesunder the time-homogeneous Markov property. For the finite-state model in part d, the Transition semigroup of a continuous-time Markov chain is

The model assumes a time-homogeneous Markov property: transition intensities depend only on the current state, not on calendar time, infection age, or earlier history. A Semi-Markov multi-state model could let recovery and death hazards depend on time since infection or entry into the current state. Weekly tests reveal recovery only by interval censoring, however, so the relevant entry and transition times are latent. Fitting the less restrictive model then requires integrating over unobserved paths, and the available data may contain too little information to identify a flexible duration-dependent hazard.

Condition on the state after $m$ steps and use the Markov property and time-homogeneous Markov property:This is the Chapman-Kolmogorov equation, or $P^{m+n}=P^m{P}^n$. A stationary distribution is a probability row vector with nonnegative entries summing to one and $\boxed{\pi P=\pi }$. Inductively $\pi P^n=\pi$, so a chain started with that distribution retains it at every time.

For the given matrix, its first stationary equation is ${\pi }_1={\pi }_1/2+({\pi }_2+{\pi }_3)/3$. Normalization gives ${\pi }_1=2/5$. The third equation gives ${\pi }_3=2{\pi }_2/3+{\pi }_3/3$, hence ${\pi }_3={\pi }_2$. ThereforeDirect multiplication verifies the second equation as well. The finite chain is irreducible, so this stationary probability distribution is unique.

For $n=0$, the conditional transition probabilities are ${\delta }_{ij}$, the entries of $P^0=I$; for $n=1$ the assertion is the definition of $P$. For the induction step, condition on the state at time $n$ and use the Markov property and time-homogeneity:The countable sum is justified by the law of total probability; all terms are nonnegative. Thus the formula follows for every $n$ by induction. It is the Chapman-Kolmogorov equation for repeated one-step transitions.

The history $\sigma (S_0,\dots ,S_n)$ is contained in $\sigma (Y_1,\dots ,Y_n)$. The increment $Y_{n+1}$ is independent of this history by the given independence of random variables. ThereforeThis depends only on the current state and not on time. It is precisely the time-homogeneous Markov property, so $S_n$ is a Markov chain with transition probabilitiesThe conditional identity is an identity that holds almost surely; histories of zero probability require no division by their probabilities.

Under the specified evolution rule, conditional on the entire history, the next event's probability distribution depends only on the current population size. This is the time-homogeneous Markov property. Birth and immigration both increase the size by one, while death decreases it by one. Writing $q=1-p$, the transition probabilities on ${\mathbb{Z}}_{\ge 0}$ areand all other entries are zero. In particular $P(0,1)=1$ and $d_0=0$. The row sum is $[kp+\theta +kq]/(k+\theta )=1$. Every neighboring pair communicates, so this is an irreducible Markov chain, specifically a birth-death-immigration jump chain.

For $a<0$, set $v_{\infty }=-{\sigma }^2/(2a)>0$. The candidate invariant probability law of a Markov process isTo prove invariance rather than merely guess a limiting probability distribution, initialize $X_t\sim \pi$ independently of future Brownian increments. Integrating the conditional characteristic function from part (i) against this centered normal distribution givesbecause $v_{\Delta }=v_{\infty }(1-e^{2a\Delta })$. The uniqueness theorem for characteristic functions implies $X_{t+\Delta }\sim \pi$ for every $\Delta \ge 0$. This proves $\pi P_{\Delta }=\pi$, as required. The time-homogeneous Markov property then also makes the process with this Ornstein-Uhlenbeck stationary initialization a strictly stationary process.

The Time-homogeneous Markov property says that the conditional law of a future state depends only on the current state and the elapsed time, not on the current calendar time or earlier history.