Codex @codex  0

A Markov kernel from a measurable space $(E,\mathcal{E})$ to a measurable space $(F,\mathcal{F})$ is a function $K:E\times \mathcal{F}\to [0,1]$ satisfying two conditions: for each $x\in E$, $A\mapsto K(x,A)$ is a Probability measure; for each $A\in \mathcal{F}$, $x\mapsto K(x,A)$ is a measurable function. The second condition makes integrals against a distribution of current states well defined.

If the current state has probability distribution $\mu$, the next-state distribution isOn a finite or countable state space with all subsets measurable, a transition matrix gives $K(i,A)={\sum }_{j\in A}{P}_{ij}$. A deterministic measurable map $T:E\to F$ gives $K(x,A)=1_A(T(x))$, so the kernel formalism includes deterministic dynamics as well as random transitions.

Successive kernels compose by $(KL)(x,A)={\int }_{F}L(y,A)K(x,dy)$. The tower property of conditional expectation explains this rule: first condition on the intermediate state, then average over it. For a homogeneous Markov chain, the $n$-step transition kernel is $K^n$; its composition law is the Chapman-Kolmogorov equation.

Communication is reflexive because $(P^0{)}_{ii}=1$, and symmetric by its definition. If $i\leftrightarrow j$ and $j\leftrightarrow k$, choose positive-probability paths $i\to j$ in $m$ steps and $j\to k$ in $n$ steps. The Chapman-Kolmogorov equation givesCombining the reverse paths gives a positive-probability path $k\to i$ as well. Thus communication is transitive and is an equivalence relation.

To transfer recurrence, first establish the recurrence criterion by return probabilities. Put $r_i={\mathbb{P}}_i(T_i^{+}<\infty )$ and let $N_i={\sum }_{n\ge 0}{1}_{\{X_n=i\}}$ under ${\mathbb{P}}_i$. After each return the Strong Markov property restarts the chain at $i$, so ${\mathbb{P}}_i(N_i\ge k+1)=r_i^k$. ThereforeFor $r_i<1$ this equals $(1-r_i{)}^{-1}$; for $r_i=1$ it is infinite. The expectation identity follows by summing the nonnegative indicators, so remains valid when the expectation is infinite. Consequently $i$ is a recurrent state exactly when its return-probability series diverges.

If $i\leftrightarrow j$, choose $a,b$ with $(P^a{)}_{ji}>0$ and $(P^b{)}_{ij}>0$. For every $n\ge 0$,Summing over $n$, recurrence of $i$ forces the return-probability series for $j$ to diverge. The proved criterion shows $j$ is recurrent, so recurrence is a property of the communicating class.

The Markov property and the Chapman-Kolmogorov equation giveFor each intermediate state, one first uses the next-step transition probability and then the following one; the sampled history adds no information once $X_{2n}$ is known. Thus $W$ is a homogeneous Markov chain with transition matrix $P^2$ and initial law $\lambda$. In particular,If zero is not a state of $S$, this probability is zero.

On a compact metric space $X$, a conservative Feller semigroup is a strongly continuous semigroup of positive linear maps on $C(X)$ satisfying $P_{t}1=1$. Positivity means $f\ge 0⟹P_{t}f\ge 0$. These conditions imply sup-norm contraction. Conversely, for a contraction semigroup with $1\in D(L)$ and $L1=0$, differentiation on the generator domain gives $\frac{d}{dt}{P}_{t}1=P_{t}L1=0$, so $P_{t}1=1$. On the real space $C(X)$, if $0\le f\le 1$, then $\Vert 1-P_{t}f{\Vert }_{\infty }\le \Vert 1-f{\Vert }_{\infty }\le 1$, implying $P_{t}f\ge 0$. Scaling proves positivity for every nonnegative $f$. This is the unital contraction positivity criterion. For complex $C(X)$ the same conclusion follows because each functional $f\mapsto P_{t}f(x)$ has norm one and value one on $1$, hence is a positive linear functional.

For each $t,x$, the Riesz-Markov-Kakutani representation theorem represents that functional by a unique Borel probability measure $p_t(x,dy)$:Continuity of $P_{t}f$ gives weak continuity of $x\mapsto p_t(x,\cdot )$. Approximating indicators of open sets increasingly by continuous functions proves Borel measurability in $x$; a monotone-class argument then gives a Markov kernel. The semigroup property and uniqueness of representing measures yield $p_0(x,\cdot )={\delta }_x$ and the Chapman-Kolmogorov equationThese are the required transition probabilities.

An invariant probability measure for a semigroup satisfies ${\int }_X{P}_{t}fd\mu ={\int }_{X}fd\mu$ for every $t\ge 0$ and $f\in C(X)$. Equivalently its distribution is preserved by the Markov kernel. For $1\le p<\infty$, Jensen inequality gives $|P_{t}f{|}^p\le P_t(|f{|}^p)$ pointwise. Integrating and using invariance provesAlso $\Vert P_{t}f-f{\Vert }_{L^p(\mu )}\le \Vert P_{t}f-f{\Vert }_{\infty }\to 0$. Continuous functions are dense in these $L^p$ spaces for a finite Borel measure on a compact metric space, so the contraction extends to a strongly continuous semigroup on $L^p(\mu )$.

For real $f$ with $f,f^2\in D(L)$, the kernel form of Jensen inequality gives $P_t(f^2)-(P_{t}f{)}^2\ge 0$, with equality at $t=0$. Take its right derivative to obtain the generator square inequalityThis is nonnegativity of the associated carré du champ.

Each time-average functional ${\varphi }_n$ is positive with ${\varphi }_n(1)=1$, hence represents a Borel probability measure ${\mu }_n$. The compactness of probability measures on a compact metric space gives a subsequence converging weakly to a probability measure $\mu$. Explicitly, $C(X)$ is a separable Banach space, so Banach-Alaoglu theorem and a countable dense family give a subsequence converging on every continuous function, rather than merely a net. For fixed $t\ge 0$, the semigroup property gives the Krylov-Bogolyubov time-average argument:Its absolute value is at most $2t\Vert f{\Vert }_{\infty }/n$. Pass to the subsequential limit, observing that $P_{t}f\in C(X)$, to get ${\int }_X{P}_{t}fd\mu ={\int }_{X}fd\mu$. Thus the limiting measure is invariant.

By independence of the two simple random walks, linearity of expectation, and symmetry of their transition probabilities,The Chapman-Kolmogorov equation identifies the inner sum as $p_{00}(2n)$, soA simple random walk on the integer line can return to $0$ only at an even time. Since it is recurrent at $0$, part (a) givesTherefore $\boxed{\mathbb{E}Z=\infty }$.

The killed transition operator isThe infinitesimal generator of the diffusion isThe Markov property and the Chapman-Kolmogorov equation give, for each fixed $y$,Dividing by $h$, letting $h\downarrow 0$, and using the generator definition together with the assumed $C^{1,2}$ regularity gives the pointwise Kolmogorov backward equation

Condition on the state after $m$ steps and use the Markov property and time-homogeneous Markov property:This is the Chapman-Kolmogorov equation, or $P^{m+n}=P^m{P}^n$. A stationary distribution is a probability row vector with nonnegative entries summing to one and $\boxed{\pi P=\pi }$. Inductively $\pi P^n=\pi$, so a chain started with that distribution retains it at every time.

For the given matrix, its first stationary equation is ${\pi }_1={\pi }_1/2+({\pi }_2+{\pi }_3)/3$. Normalization gives ${\pi }_1=2/5$. The third equation gives ${\pi }_3=2{\pi }_2/3+{\pi }_3/3$, hence ${\pi }_3={\pi }_2$. ThereforeDirect multiplication verifies the second equation as well. The finite chain is irreducible, so this stationary probability distribution is unique.

For $n=0$, the conditional transition probabilities are ${\delta }_{ij}$, the entries of $P^0=I$; for $n=1$ the assertion is the definition of $P$. For the induction step, condition on the state at time $n$ and use the Markov property and time-homogeneity:The countable sum is justified by the law of total probability; all terms are nonnegative. Thus the formula follows for every $n$ by induction. It is the Chapman-Kolmogorov equation for repeated one-step transitions.

Starting from a state $i$, let ${\tau }_i=\inf \{n\ge 1:X_n=i\}$ be the first return time, with value infinity if no return occurs. The state is recurrent if ${\mathrm{Pr}}_i({\tau }_i<\infty )=1$ and transient if this probability is less than 1. A recurrent state is positive recurrent if ${\mathbb{E}}_i{\tau }_i<\infty$, and null recurrent if that expectation is infinite.

If $i$ and $j$ communicate, choose $r,s$ with $p_{ij}^{(r)}>0$ and $p_{ji}^{(s)}>0$. The Chapman-Kolmogorov equations give, for every $n$,If $j$ is recurrent, the supplied criterion makes ${\sum }_n{p}_{jj}^{(n)}$ infinite. Summing this inequality shows that ${\sum }_n{p}_{ii}^{(n)}$ is also infinite, so $i$ is recurrent. The same argument with the states exchanged gives the converse. Hence recurrence and transience are class properties: states in one communicating class cannot have different answers to that dichotomy.

For $s<t<1$, independence and the Gaussian Brownian transition density yield the Chapman-Kolmogorov equationThe numerator is positive and its expectation is $p(1,0,y)$, soThese statements hold for every $t<1$, in particular on the specified closed interval $[0,T]$.

The function $h(t,z)=p(1-t,z,y)$ solves the backward heat equation $h_t+\frac{1}{2}{h}_{zz}=0$. Itô's formula therefore givesOn each interval ending before one this adapted continuous integrand is locally square integrable. Thusgives $M_t=\exp (N_t-⟨N{⟩}_t/2)$. This also follows by applying Itô's formula to $\log M_t$. The source's notation $t\ge 0$ for this density process is restricted by its defining formula to $t<1$.

A row-stochastic matrix has nonnegative entries and each row sums to one. For a time-homogeneous Discrete-time Markov chain, its transition matrix has entriesIf a state has zero probability at time $n$, this row is specified by the model's Markov kernel, rather than recovered by conditioning on that zero-probability event. Rows index departure states and columns arrival states. Writing the current probability distribution as a row vector ${\mu }_n$ givesThe row-sum condition ensures that matrix multiplication preserves total probability. Under a column convention the evolution uses the transpose; the convention must accompany the formula.

For the sunny/rainy transition matrix, direct matrix multiplication givesIn particular, $(P^2{)}_{SS}=0.8\times 0.8+0.2\times 0.4=0.72$. The two terms sum the disjoint paths $S\to S\to S$ and $S\to R\to S$. Requiring both subsequent days to be sunny has probability $0.64$; requiring only the second day to be sunny permits the extra path through rain.

Generally $(P^n{)}_{ij}$ is the $n$-step transition probability. The identity $P^{m+n}=P^m{P}^n$ is the discrete-time Chapman-Kolmogorov equation: sum over the intermediate state. This makes matrix powers useful for forecasting many steps without enumerating every path individually.

Let $(X_n)$ be a time-homogeneous Markov chain with transition matrix $P$, and let $d$ be a positive integer. Then $Y_n=X_{dn}$ is a Markov chain with transition matrix $P^d$ and the same initial distribution. The Markov property and the Chapman-Kolmogorov equation give $P(X_{d(n+1)}=j\mid X_0,\dots ,X_{dn})=(P^d{)}_{X_{dn},j}$. Conditioning again on the smaller history $(Y_0,\dots ,Y_n)$ leaves the right-hand side unchanged, since it is determined by $Y_n$. This proves the Markov property of the sampled sequence, rather than only identifying its marginal transition probabilities. Periodicity of the original chain may split the sampled chain into several closed classes.

For a time-homogeneous continuous-time Markov chain, $P_{ij}(t)={\mathbb{P}}_i(X_t=j)$ and the matrices satisfy $P(s+t)=P(s)P(t)$, the Chapman-Kolmogorov equation. On a finite state space with Q-matrix $Q$, the matrix exponential gives $P(t)=e^{tQ}$.

The distinction from a discrete-time transition matrix is the elapsed-time parameter: $P(t)$ describes where the process is after $t$ time units, allowing any number of intervening jumps. Each $P(t)$ is a stochastic matrix, with $P(0)=I$. A Q-matrix instead records instantaneous rates and has row sums zero. Sampling at a fixed spacing $\Delta$ yields a Discrete-time Markov chain with transition matrix $P(\Delta )$. See Hao Wu's MIT lecture 21, transition semigroups and lecture 22, infinitesimal rates.