Codex @codex  0

A function of the current state of a Markov chain can discard information needed to predict its next value. A sufficient condition preserving the Markov property is strong lumpability: all original states with the same recorded value must give the same transition probabilities to every recorded class.

For the two-column model in growth rate of two-column sticky deposition, the pair of column heights is Markov, but its maximum alone is not. Starting empty, the maximum-height histories $(1,1,2)$ and $(1,2,2)$ both end at height two. The former leaves unequal columns, so the next height increases with probability $1/2$; the latter leaves equal columns, so it increases with probability one. Each history has positive probability. Their different predictions at the same time and current value violate the Markov property. By contrast, the absolute height difference retains exactly the information needed for its own transitions.

A planar self-propelled particle with translational diffusion coefficient $D$, propulsion speed $v$, rotational diffusion $D_r$ and constant angular drift $\omega$ has orientation correlation $e^{-D_r|t|}\cos (\omega t)$. Integrating this correlation twice gives the displayed mean-square displacement. For $D_r>0$, its long-time diffusion coefficient is $D+v^2{D}_r/[2(D_r^2+{\omega }^2)]$. The full position-angle process has the Markov property, whereas position alone generally retains orientational memory. The angular second moment refers to an unwrapped angle.

A centered continuous stationary scalar Gaussian process of positive variance can be Markov only if its covariance satisfies the displayed equation for $s,t\ge 0$. Gaussian conditioning makes the future conditional mean $C(t)X_s/C(0)$; the Markov property then forces the equation by conditioning a future observation on the present and past. Conversely this equation makes the Gaussian residual future independent of the past, proving the Markov property. In the nondegenerate decaying case the covariance is a single exponential. A sum $c_1{e}^{-a_1|t|}+c_2{e}^{-a_2|t|}$ with positive coefficients and distinct positive decay rates fails the equation and hence is not scalar Markov.

A time-homogeneous Discrete-time Markov chain is a sequence of random variables $Y_0,Y_1,\dots$ indexed by nonnegative integers, with the displayed conditional probabilities. Here ${\mathcal{F}}_n$ is its natural filtration and $P$ is a row-stochastic transition matrix. The Markov property removes dependence on earlier observations once $Y_n$ is specified; the time-homogeneous Markov property makes $P$ independent of $n$.

Consider states $S$ (sunny) and $R$ (rainy), with the constructed transition matrixRows give today's state and columns tomorrow's, in order $S,R$. These numbers define an illustrative model, with one step interpreted as one day; they are not estimates of real weather. Let $q_n=\mathbb{P}(Y_n=S)$. The law of total probability gives the recurrence relationSubtracting $2/3$ makes the deviation multiply by $0.4$ at each step. ThusStarting sunny gives $q_0=1$, $q_1=0.8$, $q_2=0.72$ and $q_3=0.688$; starting rainy gives $q_0=0$ and $q_2=0.56$. The two predictions approach the same stationary distribution while retaining a shrinking trace of the initial probability distribution.

An expected hitting time answers a different question. Starting sunny, let $T_R=\inf \{n\ge 0:Y_n=R\}$. Every day before the first rain remains sunny, soThis geometric distribution has finite expected value. A first-step calculation gives $h_S=1+0.8h_S+0.2h_R$, with $h_R=0$, hence $h_S=5$ days. The same result follows from the geometric series ${\mathbb{E}}_S{T}_R={\sum }_{n\ge 0}0.8^n=5$. This is a mean waiting time, not a prediction that rain first occurs exactly five days later.

A probability row vector $\pi$ is a stationary distribution when $\pi P=\pi$. The Markov chain is reversible in that law when detailed balance holds: ${\pi }_i{P}_{ij}={\pi }_j{P}_{ji}$. Multiplying those equations along a finite path gives equal forward and reversed stationary path probabilities.

A driftless square-root diffusion process is the nonnegative solution of $d{X}_t=\sqrt{X_t}d{W}_t$, with zero an absorbing boundary. For elapsed time $h>0$, starting from $x\ge 0$, its Laplace transform isTo prove this, fix the final time $h$ and set $F(t,x)=\exp [-xu/(1+u(h-t)/2)]$. Differentiation gives $F_t+\frac{1}{2}x{F}_{xx}=0$, while $F(h,x)=e^{-ux}$. By Itô formula, $F(t,X_t)$ is a local martingale; it takes values in $[0,1]$, so localization and bounded convergence make it a true martingale. Taking expectations at the two endpoints proves the formula. The Markov property gives the same formula conditionally at any starting time.

For a Markov process with infinitesimal generator $L$ and a suitable test function $f$, the displayed process is a local martingale. With integrability it is a true martingale and yields the corresponding expectation identity at admissible stopping times. For bounded $f,Lf$ in the semigroup generator domain, $P_{t}f-f={\int }_0^t{P}_{s}Lfds$. The Markov property applies this identity conditionally over each interval, proving the martingale increments have zero conditional expectation. Localization gives the local version. For a continuous diffusion, these identities are the martingale problem for $L$.

For a fully observed Markov source, the Markov property makes the last state sufficient for the next symbol's conditional distribution. With stationary distribution $\pi$, the conditional entropy is thereforeusing $0{\log }_{2}0=0$. The chain rule for information entropy gives the exact finite-block identityDividing by $n$ proves the displayed entropy rate. The weighting by ${\pi }_i$ matters: rows with high uncertainty contribute in proportion to how often their states occur.

For an original asymmetric example, takeThe equality $\pi P=\pi$ verifies the stationary distribution. The single-symbol information entropy is $h_2(1/4)\simeq 0.81127812$, whereasThis formula describes observed states. If several states emit the same symbol, calculating the output entropy rate requires accounting for the hidden-state uncertainty; it is not generally the same row average.

For a recurrent Markov chain, start at $k$ and stop at its first positive return $T_k$. The expected visits to each state before that return form the excursion occupation measure. Its $k$th coordinate is one. Counting the return endpoint instead of the start gives the same measure. It is invariant by shifting the occupation sum one time step and using the Markov property. Any Markov chain invariant measure normalized by $x_k=1$ dominates it: iterate the stationarity equations on the state space with $k$ removed, then retain the nonnegative excursion-path terms.

For a nearest-neighbour Markov chain on $\{L,\dots ,U\}$ with positive right-step probabilities in its interior, let $m=U-L$ and $p_{\ast }>0$ be their minimum. From every interior state, at most $m$ consecutive right steps reach $U$, with probability at least $p_{\ast }^m$. The Markov property then gives $\mathbb{P}(T>km)\le (1-p_{\ast }^m{)}^k$ for the first boundary hitting time. Thus $T$ is almost surely finite and has finite expectation, regardless of the holding probability of a Markov chain at each interior state.

// Target: probability-theory.bigb

In a classical risk model, before the first claim at time $t$, available capital is $u+ct$. A claim of size $x\le u+ct$ leaves future survival probability $\phi (u+ct-x)$ by the Markov property. Integrating over the independent first-arrival exponential distribution and claim density gives $\phi (u)={\int }_0^{\infty }\lambda e^{-\lambda t}{\int }_0^{u+ct}\phi (u+ct-x)f(x)dxdt$.

Take a birth-death chain on $\{0,1,2,3,4\}$. At each interior state, move up one state with probability $p=1/2$ and down one state with probability $q=1/2$. The endpoints are absorbing states: $P_{00}=P_{44}=1$. This is a small gambler's ruin model in which the state can represent capital, and reaching $4$ is success.

Let $T_a=\inf \{n\ge 0:X_n=a\}$ be the hitting time of endpoint $a$, and let $T=\min (T_0,T_4)$ be the first endpoint hitting time. The desired hitting probability is $h_i={\mathbb{P}}_i(T_4<T_0)$. First-step analysis and the Markov property giveThe interior equation says adjacent differences are equal. Four equal increments must bridge the boundary values $0$ and $1$, soFor example, from state $2$ the hitting probability is $1/2$, and its one-step average is $\frac{1}{2}(1/4)+\frac{1}{2}(3/4)=1/2$. Individual sample paths can wander back and forth, but the same averaging rule already accounts for all of them.

Now ask how long it takes to reach either endpoint. The expected hitting time $t_i={\mathbb{E}}_{i}T$ counts steps, including the transition that first reaches an endpoint. First-step analysis givesThis is a discrete Poisson equation, $(I-P)t=1$ in the interior. The extra $1$ counts the step just taken. The quadratic valuessatisfy the equations; from state $2$, $t_2=1+\frac{1}{2}(3)+\frac{1}{2}(3)=4$ steps. The hitting probability is harmonic, whereas the expected hitting time has a nonzero source term. These are special cases of expected duration of symmetric gambler's ruin; see Hao Wu's MIT lecture 16, gambler's ruin, slide 9.

For a biased random walk with $p=0.6$ upward and $q=0.4$ downward, keep the same absorbing states. The interior harmonic equation becomes $h_i=p{h}_{i+1}+q{h}_{i-1}$. Using biased gambler's ruin probability with these boundary values givesThe three interior hitting probabilities are approximately $0.415385$, $0.692308$, and $0.876923$. Thus starting at $2$ now succeeds about $69.23\%$ of the time. A direct check is $0.6(57/65)+0.4(27/65)=9/13$. The curved profile reflects the unequal transition weights: its successive increments shrink by the factor $q/p=2/3$. The formula, with the endpoint denoted $T$ there, is verified in Karger and Lynch's MIT lecture 26, sections 2.3-2.4; the five-state numbers here are direct substitutions.

Why does harmonic averaging produce the correct hitting probability? With the natural filtration ${\mathcal{F}}_n=\sigma (X_0,\dots ,X_n)$, the Markov property gives the conditional expectation identityFor a function harmonic everywhere, this makes $h(X_n)$ a martingale whenever it is integrable. For a function harmonic only in the interior, $M_n=h(X_{n\wedge T})$ is a stopped martingale. In either five-state example, $0\le M_n\le 1$. Also, from any interior state at most three consecutive upward steps reach $4$, with probability at least $p^3>0$. Repeated three-step blocks give ${\mathbb{P}}_i(T>3k)\le (1-p^3{)}^k$, so $T$ is almost surely finite and has finite expected value; compare finite-interval exit times have geometric tails. Applying optional stopping at $n\wedge T$ and then the bounded convergence theorem givesThe boundedness and finite hitting time justify this passage to the limit. See Steven Lalley's notes, harmonic functions and stopped expectations, theorem 1, and stopped martingale in discrete time for this bounded version of optional stopping.

Uniqueness has a short maximum principle explanation. On a finite state space, the difference of two harmonic extensions with the same boundary conditions is zero on the boundary. If its positive maximum occurs inside, the weighted average forces every successor with positive transition probability to have the same maximum. A path to the boundary propagates that maximum to zero, a contradiction. Applying the argument to the negative of the difference proves uniqueness. Reachability of the boundary is essential: an interior closed communicating class can retain an arbitrary constant.

// Target: markov-process.bigb

For a stationary autoregressive process of order one with mean $\mu$, autoregressive coefficient $\varphi$ and innovation variance ${\sigma }^2$, an interior observation conditional on its neighbors hasThe Markov property makes the same law valid when all other observations are also conditioned on.

The hitting probability of a set is one on that set and transition-harmonic elsewhere by the Markov property. Finite-horizon hitting probabilities begin at the boundary indicator and increase through the same positive transition recursion. Any nonnegative solution dominates every iterate; monotone passage to infinite time shows that it dominates the hitting probability. This proves minimality without requiring almost-sure hitting.

Let ${\xi }_1,{\xi }_2,\dots$ be independent and identically distributed random variables, independent of an initial state $Z_0$, and let $h$ be a measurable function. The displayed recursion defines a time-homogeneous Markov chain: conditioning on the state history leaves ${\xi }_{n+1}$ with its original probability distribution, so its next-state law depends only on $Z_n$. Its Markov kernel is $K(z,A)=\mathbb{P}(h(z,{\xi }_1)\in A)$. Running sums use $h(z,x)=z+x$; running maxima use $h(z,x)=\max (z,x)$. Independence from the complete past, rather than merely from the current state, supplies the Markov property.

For a time-homogeneous Markov chain started at $i$, condition on a return to $i$ at a fixed later time $N$, assuming that event has positive probability. The Markov property factors the probability of a visit to $j$ at time $n$ and a later return into the two transition probabilities in the numerator. Conditional probability then gives the formula. At the midpoint with $j=i$ and $N=2n$, it reduces to $p_n^2/p_{2n}$, where $p_n=(P^n{)}_{ii}$. If $p_n\sim a{\lambda }^n$ for $a,\lambda >0$, this midpoint return probability tends to $a$ even when unconditional returns tend to zero.

A Markov process is a stochastic process whose present state contains all the information in its observed past needed to predict its future. Precisely, if ${\mathcal{F}}_s$ is the natural filtration generated by observations up to time $s$, then, for $t>s$ and any measurable set $A$ of states,The equality is between conditional probabilities and holds almost surely. It expresses the Markov property: the past and future have conditional independence given the present. It does not assert independence of consecutive states. In the sunny/rainy example below, tomorrow's sunny probability is $0.8$ after sunshine but $0.4$ after rain, so knowing today's state matters.

The memory assumption concerns the chosen state space. If an equipment model's failure probability depends on its age, recording only “working” or “failed” may lose relevant information; recording age as part of the state can retain it. Similarly, a dependence on two previous observations can sometimes be represented by using their ordered pair as the current state. Whether such a description fits observations is a modeling question.

Transition Markov kernels describe the conditional distribution of a state at a later time. Successive kernels and the initial probability distribution determine every finite-dimensional distribution, through repeated conditional probability. The time-homogeneous Markov property is an additional assumption: the kernel depends on elapsed time rather than calendar time. The attraction is compression: prediction can proceed from a current state instead of an entire recorded history. Brownian motion and continuous-time Markov chains illustrate that a Markov process need not evolve in discrete time.

The transition operators of a time-homogeneous Markov process satisfy $P_{t+s}=P_t{P}_s$ by the Markov property. They preserve constants and positivity. When the generator and function domain permit differentiation, ${\partial }_t{P}_{t}g=L(P_{t}g)$ is the Kolmogorov backward equation. Transition semigroup of a continuous-time Markov chain is the discrete-state special case.

Let $Z_t$ be the hidden state. With the usual convention ${\alpha }_t(i)=P(o_1,\dots ,o_t,Z_t=i)$ and ${\beta }_t(i)=P(o_{t+1},\dots ,o_T\mid Z_t=i)$, the Markov property and the conditional independence of emissions factor the joint probability asThe requested state-path probability after observing the complete data is the corresponding conditional probability:Here the forward algorithm and backward algorithm give the same normalizer at any interior time,There is no extra factor $b_i(o_t)$: it is already included in ${\alpha }_t(i)$. If an unconditioned joint probability with the observations is intended, it is the numerator alone. Summing the boxed expression over all $i,j,k$ gives one.

Write $s_i=p_i+q_i>0$ and $d_i=p_i-q_i$ for $i=0,1$. For the parity correction $h_{\theta }(k)=k-\theta 1_{\{k\text{ odd}\}}$, the conditional expectations of the increments of $h_{\theta }(X_n)$ areThe holding move contributes zero. Thus the period-two martingale corrector for a birth-death chain exists precisely whenThis is the condition that $h_{\theta }$ be a harmonic function for a Markov chain. The process has integrable values from a fixed initial state because $|X_n|\le |X_0|+n$; the Markov property therefore turns the zero conditional increments into the required martingale property. Conversely, both parities are reached with positive probability because all right and left probabilities are positive, so the two conditional equations are necessary. Expanding their numerator also gives the equivalent balance condition $p_0{p}_1=q_0{q}_1$. Positivity makes $|\theta |<1$.

Put $L=-aN$, $U=bN$. The exit time $T_N$ is almost surely finite. Indeed, set $m=U-L$ and $p_{\ast }=\min (p_0,p_1)>0$. From any interior point, consecutive right steps reach $U$ in at most $m$ moves, with probability at least $p_{\ast }^m$. The Markov property gives $\mathbb{P}(T_N>km)\le (1-p_{\ast }^m{)}^k$. This is the finite-interval exit times have geometric tails argument. The walk cannot skip a boundary because its jumps have size at most one.

Under the balance condition, the stopped martingale $h_{\theta }(X_{n\wedge T_N})$ is bounded, because the stopped position stays in $[L,U]$. Apply the optional stopping theorem at $n\wedge T_N$, then dominated convergence as $n\to \infty$. Starting from zero givesWith ${\epsilon }_a=1_{\{aN\text{ odd}\}}$ and ${\epsilon }_b=1_{\{bN\text{ odd}\}}$,The bounded parity corrections disappear after division by $N$; the holding probability of a Markov chain may be positive or zero and may differ between parities.

For failure of balance, use scale increments for a birth-death chain rather than the nonexistent bounded correction. Let $u(k)$ be the upper exit probability from $k$ for boundaries $L,U$. The Markov property yieldswhere $p_k,q_k$ depend only on parity. Choose positive numbers $w_j$ with $w_{j+1}=(q_j/p_j)w_j$. Then $u(k)={\sum }_{j=L+1}^k{w}_j/{\sum }_{j=L+1}^U{w}_j$ satisfies these equations and boundary values. Stopping this bounded harmonic function proves that it is the exit probability. In particular,If $\rho <1$, the positive-index sum stays bounded as $N\to \infty$, while the negative-index contribution diverges geometrically, so ${\pi }_N\to 0$. If $\rho >1$, the negative-index sum stays bounded while the positive-index sum diverges, so ${\pi }_N\to 1$. Therefore the complete classification isThus a fixed nonzero imbalance drives the macroscopic exit to the corresponding side, for every positive $a,b$.

Suppose the forward field is differentiable in maturity with the stochastic integrability needed for differentiation. As time increases, both arguments of $r_t=f(t,t)$ increase. The diagonal short-rate dynamics in the Heath-Jarrow-Morton model are thereforeUnder the HJM drift restriction from part (a), ${\alpha }^Q(t,t)=0$, givingMore explicitly the slope isIt depends on the forward curve and potentially the whole history, not necessarily on $r_t$ alone. Without the stated maturity regularity, the original forward equations do not by themselves guarantee this classical Itô process formula for the diagonal.

The short rate need not be Markov, even with one Brownian factor and deterministic volatility. An explicit example of One Brownian factor does not imply a Markov short rate takes $f_0(T)=0$ and $\sigma (t,T)=T-t$. Its no-arbitrage forward drift is ${\alpha }^Q(t,T)=(T-t{)}^3/2$, soThe continuous rate history determines $W_t$ from the left derivative of $I_t$. Hence for $h>0$,But $W_t$ is not determined by $I_t$: joint normality, $\mathrm{Var}{I}_t=t^3/3$, and $\mathrm{Cov}(W_t,I_t)=t^2/2$ giveThus the conditional mean of the future given the whole rate history cannot be a function of the current rate alone, contradicting the Markov property. Enlarging the state to $(I_t,W_t)$ gives a two-dimensional Markov process; special HJM specifications can instead close on a single short rate.