Codex @codex  0

For a finite, time-homogeneous Discrete-time Markov chain with row-stochastic transition matrix $P$, a state $a$ is absorbing exactly when $P_{aa}=1$. Nonnegativity and the row sum then force $P_{aj}=0$ for $j\ne a$: once entered, $a$ is never left. Equivalently, $\{a\}$ is a closed communicating class. A larger closed communicating class may keep moving internally; closure means confinement to a set, whereas absorption at a state means remaining at one point.

An absorbing state is useful for recording a terminal outcome: after a toy routine finishes, keep its outcome in the state space so that the same transition matrix still describes subsequent steps. Calling a finite Markov chain absorbing usually also requires that some absorbing state be accessible from every state. Merely containing an absorbing state is insufficient: another closed communicating class could prevent absorption. See Grinstead and Snell, Introduction to Probability, section 11.2 for this convention.

Consider a constructed routine with unfinished modes $u,v$ and terminal outcomes $A,B$. All the following transition probabilities are chosen for illustration, not estimated from observations. In state order $(u,v,A,B)$, takeThe communicating class $\{u,v\}$ is open, while $\{A\}$ and $\{B\}$ are closed communicating classes consisting of absorbing states. Both terminal outcomes are accessible from both unfinished modes, sometimes by an indirect path.

Let $T=\inf \{n\ge 0:X_n\in \{A,B\}\}$ be the terminal hitting time. Write $h_i={\mathbb{P}}_i(X_T=A)$ for the absorption probability and $t_i={\mathbb{E}}_{i}T$ for the expected hitting time, with the subscript denoting the starting state. First-step analysis givesHere $h_A=1$, $h_B=0$, and $t_A=t_B=0$. Solving these two pairs of linear equations yieldsStarting at $u$ therefore ends at $A$ with probability $0.6$ and takes $3.2$ steps on average to reach either terminal state. The terminal hitting time counts the transition into the outcome, but no later self-transitions. Outcome and duration answer different questions: from $v$, outcome $A$ is less likely, yet termination is faster on average.

The matrix power $P^2$ gives the probability distribution after two steps from $u$ asThus ${\mathbb{P}}_u(T>2)=1/2$, and the probability of having already reached $A$ is $3/8$, still below its eventual absorption probability $3/5$. From either unfinished mode, each next step has terminal probability at least $1/4$. Consequently ${\mathbb{P}}_i(T>n)\le (3/4{)}^n$, proving termination almost surely and finite expected hitting time directly. The Fundamental matrix of a finite absorbing Markov chain packages both calculations and counts visits to unfinished modes.

Build a candidate incrementally, abandoning a partial candidate as soon as no valid completion is possible. A depth-first search explores one choice, restores the previous state, and tries the next. Constraint propagation and bounds can greatly reduce the visited state space, although the worst-case time complexity may remain exponential.

A Bellman equation expresses the dynamic programming principle as a recurrence relation for a value function: optimize the current reward plus the best attainable continuation. Let $s$ be a dynamic programming state, $A_t(s)$ the allowed actions at time $t$, $r_t(s,a)$ the immediate reward, and $F_t(s,a)$ the next dynamic programming state. With terminal time $T$ and terminal reward $g$, the deterministic finite-horizon Bellman equation isHere $V_t(s)$ measures total reward from time $t$ through the terminal reward. Start with $g$ and evaluate backwards; the terminal condition anchors the recurrence relation. For finite nonempty action sets, finite rewards, and finite terminal rewards, a maximum exists at each step. If an optimum need not be attained, use a supremum. A minimization problem instead uses a minimum or infimum. The 0-1 knapsack problem above is the same comparison between an immediate item value plus a continuation and the continuation after skipping. See Bertsekas, MIT lecture 2: the backwards recursion.

In a Markov decision process, the next dynamic programming state $S_{t+1}$ is random. If $r_t(s,a)$ denotes the expected immediate reward and $\beta$ is a discount factor, the finite-horizon Bellman equation becomesThe conditional expectation averages over the next dynamic programming state before choosing the current action; that action cannot use a future outcome that has not yet been observed. For a finite state space with transition probabilities $p_t(s^{\prime }\mid s,a)$, the continuation term is $\beta {\sum }_{s^{\prime }}{p}_t(s^{\prime }\mid s,a)V_{t+1}(s^{\prime })$. Taking $\beta =1$ gives the undiscounted finite-horizon problem. With a discount factor $0\le \beta <1$, rewards $k$ steps ahead carry a factor ${\beta }^k$, including the terminal reward. A maximizing action for each $(t,s)$ specifies a Markov policy, which contains decisions rather than just optimal scores.

For an infinite horizon with a finite state space, finite nonempty action sets, time-independent transition probabilities, bounded rewards $|r(s,a)|\le M$, and a discount factor $0\le \beta <1$, the optimal value function satisfiesIn this setting the maximizing dynamic programming operator is a contraction mapping in the supremum norm, so the Banach fixed-point theorem gives a unique bounded fixed point and convergence of repeated application to it. See Bertsekas, MIT lecture 15: bounded discounted problems. These hypotheses matter: with one state, one action, zero reward, and $\beta =1$, the Bellman equation is merely $V(s)=V(s)$, so every finite constant is a fixed point, although the actual total-reward value function is zero. The average-reward Bellman equation and Hamilton-Jacobi-Bellman equation address other settings with their own conditions.

The butterfly effect is the intuitive idea that an arbitrarily small change in an initial condition can eventually change the observed future substantially. In dynamical systems, sensitive dependence makes this precise for a map $F:X\to X$ on a metric space $(X,d)$: there is one separation scale $\delta >0$ such thatThe same $\delta$ works at every point, whereas $y$ and the separation time $n$ may depend on the point and the requested closeness. For a flow map, replace the iteration of a map $F^n$ by ${\Phi }_t$ with some $t\ge 0$. The state space matters: restricting to a single fixed point removes the alternative nearby states needed for sensitivity.

Sensitivity is compatible with deterministic evolution. Given an exact initial condition, there is still one orbit; uncertainty concerns which exact state a measurement represents. It also allows accurate short-term prediction. For a continuous function $F$, each fixed finite iterated function $F^n$ is continuous, so sufficiently close initial states remain close at any specified finite step. What fails is a single initial accuracy that controls every future step.

The logistic map illustrates both sides. At $r=2.8$, nearby orbits in the basin of attraction of its attracting fixed point approach one another. At $r=4$, arbitrarily close states can eventually separate by a macroscopic fraction of $[0,1]$. The definition asserts the existence of separating neighbours, not that every nearby pair must separate: $x$ and $1-x$ have exactly the same next state under the logistic map and hence the same subsequent orbit. Nor must separation increase at every step.

A positive Lyapunov exponent describes average exponential growth of an infinitesimal displacement along an orbit, when the relevant limit exists. It is related to sensitivity but is not a substitute for its uniform definition. For a differentiable interval map, the chain rule givesAt $r=4$, the logistic map has $\lambda =\log 2$ for Lebesgue measure-almost every initial condition, while exceptional orbits can have different exponents. Finite-separation growth is only approximately exponential while the linear approximation remains useful. In a bounded state space, a separation cannot grow exponentially forever.

Sensitivity alone is also weaker than Devaney chaos, which additionally requires topological transitivity and dense periodic points. For example, $F(x)=2x$ on the real line has sensitive dependence and a positive Lyapunov exponent, but its only periodic point is zero, so it is not Devaney-chaotic. This distinguishes growth of uncertainty from the richer organization of recurrent orbits.

Edward N. Lorenz's "Deterministic Nonperiodic Flow" (1963) studied a thermal convection model with three ordinary differential equations. Its numerical orbits exhibited persistent nonperiodic motion and instability to small changes of the initial condition. Lorenz discussed the consequences for very-long-range weather forecasting: a deterministic law need not make a detailed distant future reliably recoverable from approximate observations. The historically relevant lesson of the Lorenz system is this growth of uncertainty under deterministic evolution.^[ref]

Every square matrix satisfies its own characteristic polynomial. More explicitly, for an $n$-by-$n$ matrix $A$ over a commutative ring with identity, formThenHere $0$ is the zero matrix, and the constant term multiplies the identity matrix. This is a polynomial evaluated at a matrix; it is stronger than saying that each eigenvalue is a root of a polynomial. Neither diagonalizability nor distinct eigenvalues is required.^[ref]

The practical payoff is that arbitrarily high matrix powers can be reduced to a combination of $I,A,\dots ,A^{n-1}$. These powers arise when repeatedly applying a linear map, solving a linear recurrence relation, or evolving a finite state space model. The characteristic polynomial turns that repeated action into a fixed-order scalar recurrence. The minimal polynomial may reduce the number of required powers further.

In a finite, time-homogeneous Discrete-time Markov chain with row-stochastic transition matrix $P$, write $i\to j$ when $j$ is accessible from $i$, meaning $(P^n{)}_{ij}>0$ for some $n\ge 0$. States communicate, written $i\leftrightarrow j$, when both $i\to j$ and $j\to i$. Communication is an equivalence relation, and a communicating class is an entire equivalence classThus communicating classes form a partition of the state space; a smaller mutually communicating subset is not itself a class. See Maria Cameron's Markov Chains notes, section 1.2.

The allowance of zero steps matters: $P^0=I$, the identity matrix, makes $i\to i$ true even for a state with no possible positive-time return. Symmetry is built into communication. For transitivity, concatenate a positive-probability path from $i$ to $j$ with one from $j$ to $k$, and do the same in reverse. These observations prove the equivalence relation directly. One-way accessibility alone is not symmetric and does not define the classes; see Accessible state of a Markov chain.

A closed communicating class satisfies $P_{ij}=0$ for every $i\in C$ and $j\notin C$. Once entered, it cannot be left. A singleton class is an absorbing state exactly when it is closed. An open communicating class has an outgoing transition; after a path leaves it, it cannot return, since that would make the visited outside states communicate with it. The internal motion of a closed class can nevertheless continue forever.

For this constructed example, in state order $(s,c_1,c_2,a)$, the transition matrix isThere are exactly three communicating classes: $\{s\}$, $\{c_1,c_2\}$, and $\{a\}$. Only the first is open. The second is closed but neither of its states is an absorbing state; they alternate deterministically. Although $s\to c_2$ via $c_1$, the reverse accessibility fails. A direct matrix power check gives $(P^2{)}_{s,c_2}=1/4$ and $(P^2{)}_{c_2,s}=0$.

Starting at $s$, let $\tau$ be the first hitting time of either closed class. Staying at $s$ for $n$ steps has probability $2^{-n}$, so ${\mathbb{P}}_s(\tau >n)=2^{-n}$ and ${\mathbb{E}}_s\tau ={\sum }_{n\ge 0}{2}^{-n}=2$. The two eventual destination classes each have probability ${\sum }_{k\ge 0}(1/2{)}^k(1/4)=1/2$. Hence the hitting probability of the absorbing state $a$ is only $1/2$, and its unconditional expected hitting time from $s$ is infinite: on half the paths it is never hit. In particular, entering a closed class with certainty does not imply entering an absorbing state with certainty.

Classifying the state space separates two questions: which closed region will eventually be entered, and how does the Markov chain move inside that region? In this example the restriction to $\{c_1,c_2\}$ is an irreducible Markov chain with period two; the restriction to $\{a\}$ is a one-state irreducible Markov chain with period one. The whole Markov chain is reducible. The absorbing state example illustrates a different possibility: every closed class is a singleton, so eventual class selection is also terminal outcome selection.

Controllability asks whether the available control inputs can move a dynamical system between every pair of states in its declared state space. Fixing a duration $T>0$ gives the precise condition ${\mathcal{R}}_T(x_0)=X$ for every $x_0\in X$, where $X$ is the state space and ${\mathcal{R}}_T(x_0)$ its reachable set. With freely chosen duration, every pair must instead be connectable in some permitted finite time, which may depend on the pair. Always specify admissible control inputs, deadlines and any constraints along the path. ^[ref]

The surprising part is that one control input can control several state variables by acting at different times. For the double integrator, acceleration changes velocity directly, while the accumulated velocity changes position. Starting from rest, accelerate during the first half of the journey and brake during the second: the final velocity can vanish although the final position has changed. The figure is an analytic mathematical example in dimensionless quantities, with no experimental measurements.

Controllability is useful before trying to design a controller: a requested endpoint outside the reachable set cannot be attained by clever choice of feedback control. The Kalman rank criterion makes this an exact linear algebra calculation for the stated linear time-invariant system model. It concerns whole states, so attaining one plant output is not enough to establish controllability. In quantum control, the declared state space may be an orbit constrained by conservation laws; see quantum controllability and unitary orbit of a density operator.

A dynamical system consists of a state space together with a rule that determines how its state evolves in time.

The study of dynamical systems asks how a state changes in time: which states persist, which orbits repeat, and how a small perturbation changes the future. A dynamical system combines a state space with an evolution rule. For a seasonal population, the state might be its abundance and one step one generation; for a harmonic oscillator, a state vector includes both position and velocity. Choosing a sufficient state is essential: position alone does not determine where an oscillator moves next.

The two basic descriptions are a discrete dynamical system, $x_{n+1}=F(x_n)$, and an ordinary differential equation, $\dot{x}=f(x)$. Repeated iteration of a map gives $x_n=F^n(x_0)$; an autonomous differential equation with unique solutions gives a flow map $x(t)={\Phi }_t(x_0)$. In either case, evolving for two successive durations agrees with evolving for their sum. The logistic map provides a particularly simple discrete dynamical system; the Lorenz system supplies a continuous-time example with a very different geometry.

Deterministic evolution says that the exact rule and exact initial condition specify one future. Predicting that future from measurements is a separate question. An instrument supplies a range of possible initial conditions, and a computation adds small errors. Near an asymptotically stable fixed point, nearby orbits converge and the long-term state can become easier to predict. With sensitive dependence on initial conditions, arbitrarily close starting states can eventually lead to a visible disagreement, even though each follows the same rule.

Much of the usefulness of dynamical systems comes from describing outcomes without writing every solution explicitly. Fixed points identify possible resting states; periodic orbits identify repeating behaviour; stability theory asks whether those behaviours survive small disturbances. A bifurcation diagram shows how changing a bifurcation parameter changes the possible outcomes. This is useful in population dynamics, where a stable abundance and an alternating abundance have different consequences, and in thermal convection, where a steady flow and a changing flow transport heat differently.

The beauty of the subject is that a small formula can organize a large variety of behaviour. The logistic map moves from an attracting fixed point to attracting periodic orbits and then to parameter values supporting sensitive motion. Its period-doubling cascade has a recognizable structure across many unimodal maps. Nevertheless, Devaney chaos, Glendinning chaos, and a positive Lyapunov exponent express different properties; a claim of chaos should specify its definition, its state space, and the parameter values under consideration.

A dynamic programming state is an element of a state space that summarizes all information needed for a remaining mathematical optimization problem. Histories can share a value function entry only when their allowed continuations and future rewards agree, or have the same conditional laws in a stochastic control problem. Time or the remaining item index may be part of the summary. If past decisions still affect feasibility or rewards, enlarge the dynamic programming state; forgetting those effects invalidates the Bellman equation.

Take a birth-death chain on $\{0,1,2,3,4\}$. At each interior state, move up one state with probability $p=1/2$ and down one state with probability $q=1/2$. The endpoints are absorbing states: $P_{00}=P_{44}=1$. This is a small gambler's ruin model in which the state can represent capital, and reaching $4$ is success.

Let $T_a=\inf \{n\ge 0:X_n=a\}$ be the hitting time of endpoint $a$, and let $T=\min (T_0,T_4)$ be the first endpoint hitting time. The desired hitting probability is $h_i={\mathbb{P}}_i(T_4<T_0)$. First-step analysis and the Markov property giveThe interior equation says adjacent differences are equal. Four equal increments must bridge the boundary values $0$ and $1$, soFor example, from state $2$ the hitting probability is $1/2$, and its one-step average is $\frac{1}{2}(1/4)+\frac{1}{2}(3/4)=1/2$. Individual sample paths can wander back and forth, but the same averaging rule already accounts for all of them.

Now ask how long it takes to reach either endpoint. The expected hitting time $t_i={\mathbb{E}}_{i}T$ counts steps, including the transition that first reaches an endpoint. First-step analysis givesThis is a discrete Poisson equation, $(I-P)t=1$ in the interior. The extra $1$ counts the step just taken. The quadratic valuessatisfy the equations; from state $2$, $t_2=1+\frac{1}{2}(3)+\frac{1}{2}(3)=4$ steps. The hitting probability is harmonic, whereas the expected hitting time has a nonzero source term. These are special cases of expected duration of symmetric gambler's ruin; see Hao Wu's MIT lecture 16, gambler's ruin, slide 9.

For a biased random walk with $p=0.6$ upward and $q=0.4$ downward, keep the same absorbing states. The interior harmonic equation becomes $h_i=p{h}_{i+1}+q{h}_{i-1}$. Using biased gambler's ruin probability with these boundary values givesThe three interior hitting probabilities are approximately $0.415385$, $0.692308$, and $0.876923$. Thus starting at $2$ now succeeds about $69.23\%$ of the time. A direct check is $0.6(57/65)+0.4(27/65)=9/13$. The curved profile reflects the unequal transition weights: its successive increments shrink by the factor $q/p=2/3$. The formula, with the endpoint denoted $T$ there, is verified in Karger and Lynch's MIT lecture 26, sections 2.3-2.4; the five-state numbers here are direct substitutions.

Why does harmonic averaging produce the correct hitting probability? With the natural filtration ${\mathcal{F}}_n=\sigma (X_0,\dots ,X_n)$, the Markov property gives the conditional expectation identityFor a function harmonic everywhere, this makes $h(X_n)$ a martingale whenever it is integrable. For a function harmonic only in the interior, $M_n=h(X_{n\wedge T})$ is a stopped martingale. In either five-state example, $0\le M_n\le 1$. Also, from any interior state at most three consecutive upward steps reach $4$, with probability at least $p^3>0$. Repeated three-step blocks give ${\mathbb{P}}_i(T>3k)\le (1-p^3{)}^k$, so $T$ is almost surely finite and has finite expected value; compare finite-interval exit times have geometric tails. Applying optional stopping at $n\wedge T$ and then the bounded convergence theorem givesThe boundedness and finite hitting time justify this passage to the limit. See Steven Lalley's notes, harmonic functions and stopped expectations, theorem 1, and stopped martingale in discrete time for this bounded version of optional stopping.

Uniqueness has a short maximum principle explanation. On a finite state space, the difference of two harmonic extensions with the same boundary conditions is zero on the boundary. If its positive maximum occurs inside, the weighted average forces every successor with positive transition probability to have the same maximum. A path to the boundary propagates that maximum to zero, a contradiction. Applying the argument to the negative of the difference proves uniqueness. Reachability of the boundary is essential: an interior closed communicating class can retain an arbitrary constant.

// Target: markov-process.bigb

On a finite state space $F$ with finite objective function values, the fixed-$T$ target is a Boltzmann distribution:The partition function $Z_T$ normalizes the weights but cancels from the Metropolis–Hastings acceptance probability. A short detailed balance proof explains the target. With symmetric $q$ and $x\ne y$,Summing over $x$ gives ${\pi }_T{P}_T={\pi }_T$, so this is a stationary distribution. On a finite state space, a connected graph of positive symmetric proposals makes the accepted Markov chain an irreducible Markov chain. Convergence from any starting state additionally requires an aperiodic Markov chain; one positive self-transition suffices under irreducibility. Replacing $q$ by $(I+q)/2$ ensures this. For asymmetric $q$, require connectivity through bidirectional proposals: one-way proposals are rejected.^[ref]

In the five-state barrier example for simulated annealing, $T_{\ast }=1/\ln 2$ gives exact Boltzmann weights $2^{-E(x)}$. Multiplying by $16$ givesFor the uphill edge, detailed balance is visible numerically:Thus the global minimum has the largest individual stationary probability, $16/33$, but is far from certain at this positive $T_{\ast }$. At $T_{\ast }/2$ the exact stationary distribution is $(16,64,1,16,256)/353$.

For any finite $F$, let $M$ be the set of global minima and $E_{\min }$ the smallest objective function value. Cancelling $e^{-E_{\min }/T}$ from numerator and denominator shows that each nonminimal state's weight tends to zero as $T\downarrow 0$, while every state in $M$ has equal weight. Therefore ${\pi }_T$ tends to the uniform distribution on $M$. This is a statement about a family of equilibrium probability distributions; it does not prove that a cooling Markov chain follows that family.

Each emitted symbol identifies the state of a stationary finite Markov chain. Here the alphabet is the state space itself. Its transition matrix $P$ and stationary distribution $\pi$ determine the entropy rate of a fully observed Markov source.

A harmonic function assigns a value to each state of a Markov chain such that one transition leaves its expected value unchanged. This turns a question about many possible future sample paths, such as which endpoint is reached first, into a local averaging rule. For a transition matrix $P$, a real-valued function $h$ is harmonic at state $i$ whenHere ${\mathbb{E}}_i$ means expected value conditional on starting at $X_0=i$; on a countable state space the sum must be absolutely convergent. If this holds at every state, write $Ph=h$. At an absorbing state the equation is automatic, so prescribed boundary conditions supply information that the averaging rule cannot.

For a finite Markov chain, choose a boundary set $B$ that can be reached from every state, and assign boundary values $b$. If $T_B$ is its first hitting time, the harmonic extension isChoosing $b$ to be the indicator function of one target within $B$ gives a hitting probability. This is a discrete Dirichlet problem: enforce the averaging rule outside $B$ and fix the values on $B$. See Hao Wu's MIT lecture 8, harmonic extension, slide 5. The construction extends the mean-value idea of a harmonic function; for a random walk on a graph it is a discrete harmonic function.

For a finite, time-homogeneous Discrete-time Markov chain with row-stochastic transition matrix $P$, irreducibility meansEquivalently, all states form one communicating class. The step count may depend on $i,j$; irreducibility does not require every one-step transition probability to be positive. Drawing an arrow $i\to j$ whenever $P_{ij}>0$ makes the condition exactly that of a strongly connected directed graph. It is a property of the entire chosen state space, independent of the initial probability distribution.

Irreducibility describes where the Markov chain can travel; aperiodicity describes the possible lengths of return paths. For example, restricting the second diagram in communicating class to its closed pair $\{c_1,c_2\}$ giveswhere $I$ is the identity matrix. This Markov chain is irreducible, but each state's period of a state in a Markov chain is two. Every destination is reachable, yet a given destination cannot be reached at every sufficiently large step count. Starting at $c_1$ produces alternating one-time probability distributions. See Hao Wu's MIT lecture 2, irreducibility and periods for the definitions.

Adding a holding probability of a Markov chain $\epsilon$ at each state, with $0<\epsilon <1$, givesThe arrows between the two states remain, so the Markov chain remains irreducible. A positive one-step return makes it an aperiodic Markov chain. At $\epsilon =1/2$, both rows are $(1/2,1/2)$ and $C_{1/2}^n=C_{1/2}$ for every $n\ge 1$: the initial state is forgotten after one step. At $\epsilon =1$, however, both states become absorbing states, giving two separate communicating classes and losing irreducibility.

For a finite Markov chain, irreducibility guarantees a unique stationary distribution with positive weight at every state. Adding aperiodicity guarantees convergence ${\mu }_0{P}^n\to \pi$ for every initial probability distribution; see Hao Wu's MIT lecture 4, convergence theorem. The Markov chain ergodic theorem gives convergence of empirical visit frequencies even for an irreducible periodic Markov chain. These results explain why checking connectivity is a useful first step before studying equilibrium. Conversely, any absorbing state prevents irreducibility if there is another state: it cannot reach that state. The one-state Markov chain is the exception.

Local time records how much a stochastic process visits a level. Its normalization depends on the process: when the state space is a discrete space, it is occupation time at a state, while local time of a semimartingale is an occupation density weighted by quadratic variation. Brownian local time is a special case.

The term Markov chain commonly denotes a Markov process on a finite or countable state space. Time may be discrete, as for a daily record, or continuous, as for a continuous-time Markov chain. Some authors also use “Markov chain” for discrete-time processes on more general state spaces; specifying both time and state space avoids this ambiguity.

For a Discrete-time Markov chain, its initial probability distribution ${\mu }_0$ and transition matrix $P$ specify the probability of each finite path:This is the Markov factorization, obtained by applying conditional probability repeatedly. See Hao Wu's MIT lecture 1, definition and distribution evolution.

For the constructed sunny/rainy Markov chain below, starting sunny makes the path sunny, rainy, sunny have probability $1\times 0.2\times 0.4=0.08$. Starting instead with stationary distribution $(2/3,1/3)$ makes that path's probability $(2/3)\times 0.2\times 0.4=4/75$. The transition probabilities are identical in both experiments; the initial probability distribution distinguishes them.

A Markov process is a stochastic process whose present state contains all the information in its observed past needed to predict its future. Precisely, if ${\mathcal{F}}_s$ is the natural filtration generated by observations up to time $s$, then, for $t>s$ and any measurable set $A$ of states,The equality is between conditional probabilities and holds almost surely. It expresses the Markov property: the past and future have conditional independence given the present. It does not assert independence of consecutive states. In the sunny/rainy example below, tomorrow's sunny probability is $0.8$ after sunshine but $0.4$ after rain, so knowing today's state matters.

The memory assumption concerns the chosen state space. If an equipment model's failure probability depends on its age, recording only “working” or “failed” may lose relevant information; recording age as part of the state can retain it. Similarly, a dependence on two previous observations can sometimes be represented by using their ordered pair as the current state. Whether such a description fits observations is a modeling question.

Transition Markov kernels describe the conditional distribution of a state at a later time. Successive kernels and the initial probability distribution determine every finite-dimensional distribution, through repeated conditional probability. The time-homogeneous Markov property is an additional assumption: the kernel depends on elapsed time rather than calendar time. The attraction is compression: prediction can proceed from a current state instead of an entire recorded history. Brownian motion and continuous-time Markov chains illustrate that a Markov process need not evolve in discrete time.

Two independently available parts of a thermodynamic system are noninteracting in the canonical description when their state space is a Cartesian product and their energy is additive, $E(a,b)=E_A(a)+E_B(b)$. Both conditions are needed for the canonical partition function to factor:An additive expression on a constrained set of allowed state pairs need not give this factorization. Relabeling the states to make their allowed labels independent can instead introduce an interaction into the energy. For two two-state labels, an additive energy necessarily satisfies the rectangular identity $E_{11}+E_{22}=E_{12}+E_{21}$.

The convention intended by this question is existence of a dense forward orbit, also called point transitivity. On a nonempty compact metric space, this means that some $x\in X$ satisfiesEquivalently, that forward orbit visits every nonempty open set.

The other common definition of topological transitivity uses intersections of iterated open sets, as in part (b). The distinction matters on spaces with isolated points: parts (b) and (c) ask precisely how the two conventions are related. We use point transitivity for the dense-orbit convention throughout this solution. As usual for this definition, the state space is assumed nonempty; on an empty space the open-set condition is vacuous and there is no point with a forward orbit.