Codex @codex  0

For a finite, time-homogeneous Discrete-time Markov chain with row-stochastic transition matrix $P$, a state $a$ is absorbing exactly when $P_{aa}=1$. Nonnegativity and the row sum then force $P_{aj}=0$ for $j\ne a$: once entered, $a$ is never left. Equivalently, $\{a\}$ is a closed communicating class. A larger closed communicating class may keep moving internally; closure means confinement to a set, whereas absorption at a state means remaining at one point.

An absorbing state is useful for recording a terminal outcome: after a toy routine finishes, keep its outcome in the state space so that the same transition matrix still describes subsequent steps. Calling a finite Markov chain absorbing usually also requires that some absorbing state be accessible from every state. Merely containing an absorbing state is insufficient: another closed communicating class could prevent absorption. See Grinstead and Snell, Introduction to Probability, section 11.2 for this convention.

Consider a constructed routine with unfinished modes $u,v$ and terminal outcomes $A,B$. All the following transition probabilities are chosen for illustration, not estimated from observations. In state order $(u,v,A,B)$, takeThe communicating class $\{u,v\}$ is open, while $\{A\}$ and $\{B\}$ are closed communicating classes consisting of absorbing states. Both terminal outcomes are accessible from both unfinished modes, sometimes by an indirect path.

Let $T=\inf \{n\ge 0:X_n\in \{A,B\}\}$ be the terminal hitting time. Write $h_i={\mathbb{P}}_i(X_T=A)$ for the absorption probability and $t_i={\mathbb{E}}_{i}T$ for the expected hitting time, with the subscript denoting the starting state. First-step analysis givesHere $h_A=1$, $h_B=0$, and $t_A=t_B=0$. Solving these two pairs of linear equations yieldsStarting at $u$ therefore ends at $A$ with probability $0.6$ and takes $3.2$ steps on average to reach either terminal state. The terminal hitting time counts the transition into the outcome, but no later self-transitions. Outcome and duration answer different questions: from $v$, outcome $A$ is less likely, yet termination is faster on average.

The matrix power $P^2$ gives the probability distribution after two steps from $u$ asThus ${\mathbb{P}}_u(T>2)=1/2$, and the probability of having already reached $A$ is $3/8$, still below its eventual absorption probability $3/5$. From either unfinished mode, each next step has terminal probability at least $1/4$. Consequently ${\mathbb{P}}_i(T>n)\le (3/4{)}^n$, proving termination almost surely and finite expected hitting time directly. The Fundamental matrix of a finite absorbing Markov chain packages both calculations and counts visits to unfinished modes.

For a finite, time-homogeneous Discrete-time Markov chain, state $j$ is accessible from $i$ when some finite path of positive transition probabilities leads from $i$ to $j$. Equivalently, $(P^n{)}_{ij}>0$ for some $n\ge 0$, allowing a zero-step path when $i=j$. Accessibility is reflexive and transitive; communicating states require accessibility in both directions. See Ward Whitt's lecture notes of September 24, 2013, classification of states. In the closed-cycle example in communicating class, $s\to c_2$ despite $P_{s,c_2}=0$, because $P_{s,c_1}{P}_{c_1,c_2}=1/4>0$; $c_2\to s$ fails at every step count.

In a finite, time-homogeneous Discrete-time Markov chain with row-stochastic transition matrix $P$, write $i\to j$ when $j$ is accessible from $i$, meaning $(P^n{)}_{ij}>0$ for some $n\ge 0$. States communicate, written $i\leftrightarrow j$, when both $i\to j$ and $j\to i$. Communication is an equivalence relation, and a communicating class is an entire equivalence classThus communicating classes form a partition of the state space; a smaller mutually communicating subset is not itself a class. See Maria Cameron's Markov Chains notes, section 1.2.

The allowance of zero steps matters: $P^0=I$, the identity matrix, makes $i\to i$ true even for a state with no possible positive-time return. Symmetry is built into communication. For transitivity, concatenate a positive-probability path from $i$ to $j$ with one from $j$ to $k$, and do the same in reverse. These observations prove the equivalence relation directly. One-way accessibility alone is not symmetric and does not define the classes; see Accessible state of a Markov chain.

A closed communicating class satisfies $P_{ij}=0$ for every $i\in C$ and $j\notin C$. Once entered, it cannot be left. A singleton class is an absorbing state exactly when it is closed. An open communicating class has an outgoing transition; after a path leaves it, it cannot return, since that would make the visited outside states communicate with it. The internal motion of a closed class can nevertheless continue forever.

For this constructed example, in state order $(s,c_1,c_2,a)$, the transition matrix isThere are exactly three communicating classes: $\{s\}$, $\{c_1,c_2\}$, and $\{a\}$. Only the first is open. The second is closed but neither of its states is an absorbing state; they alternate deterministically. Although $s\to c_2$ via $c_1$, the reverse accessibility fails. A direct matrix power check gives $(P^2{)}_{s,c_2}=1/4$ and $(P^2{)}_{c_2,s}=0$.

Starting at $s$, let $\tau$ be the first hitting time of either closed class. Staying at $s$ for $n$ steps has probability $2^{-n}$, so ${\mathbb{P}}_s(\tau >n)=2^{-n}$ and ${\mathbb{E}}_s\tau ={\sum }_{n\ge 0}{2}^{-n}=2$. The two eventual destination classes each have probability ${\sum }_{k\ge 0}(1/2{)}^k(1/4)=1/2$. Hence the hitting probability of the absorbing state $a$ is only $1/2$, and its unconditional expected hitting time from $s$ is infinite: on half the paths it is never hit. In particular, entering a closed class with certainty does not imply entering an absorbing state with certainty.

Classifying the state space separates two questions: which closed region will eventually be entered, and how does the Markov chain move inside that region? In this example the restriction to $\{c_1,c_2\}$ is an irreducible Markov chain with period two; the restriction to $\{a\}$ is a one-state irreducible Markov chain with period one. The whole Markov chain is reducible. The absorbing state example illustrates a different possibility: every closed class is a singleton, so eventual class selection is also terminal outcome selection.

A time-homogeneous Discrete-time Markov chain is a sequence of random variables $Y_0,Y_1,\dots$ indexed by nonnegative integers, with the displayed conditional probabilities. Here ${\mathcal{F}}_n$ is its natural filtration and $P$ is a row-stochastic transition matrix. The Markov property removes dependence on earlier observations once $Y_n$ is specified; the time-homogeneous Markov property makes $P$ independent of $n$.

Consider states $S$ (sunny) and $R$ (rainy), with the constructed transition matrixRows give today's state and columns tomorrow's, in order $S,R$. These numbers define an illustrative model, with one step interpreted as one day; they are not estimates of real weather. Let $q_n=\mathbb{P}(Y_n=S)$. The law of total probability gives the recurrence relationSubtracting $2/3$ makes the deviation multiply by $0.4$ at each step. ThusStarting sunny gives $q_0=1$, $q_1=0.8$, $q_2=0.72$ and $q_3=0.688$; starting rainy gives $q_0=0$ and $q_2=0.56$. The two predictions approach the same stationary distribution while retaining a shrinking trace of the initial probability distribution.

An expected hitting time answers a different question. Starting sunny, let $T_R=\inf \{n\ge 0:Y_n=R\}$. Every day before the first rain remains sunny, soThis geometric distribution has finite expected value. A first-step calculation gives $h_S=1+0.8h_S+0.2h_R$, with $h_R=0$, hence $h_S=5$ days. The same result follows from the geometric series ${\mathbb{E}}_S{T}_R={\sum }_{n\ge 0}0.8^n=5$. This is a mean waiting time, not a prediction that rain first occurs exactly five days later.

A probability row vector $\pi$ is a stationary distribution when $\pi P=\pi$. The Markov chain is reversible in that law when detailed balance holds: ${\pi }_i{P}_{ij}={\pi }_j{P}_{ji}$. Multiplying those equations along a finite path gives equal forward and reversed stationary path probabilities.

For a finite, time-homogeneous Discrete-time Markov chain with row-stochastic transition matrix $P$, irreducibility meansEquivalently, all states form one communicating class. The step count may depend on $i,j$; irreducibility does not require every one-step transition probability to be positive. Drawing an arrow $i\to j$ whenever $P_{ij}>0$ makes the condition exactly that of a strongly connected directed graph. It is a property of the entire chosen state space, independent of the initial probability distribution.

Irreducibility describes where the Markov chain can travel; aperiodicity describes the possible lengths of return paths. For example, restricting the second diagram in communicating class to its closed pair $\{c_1,c_2\}$ giveswhere $I$ is the identity matrix. This Markov chain is irreducible, but each state's period of a state in a Markov chain is two. Every destination is reachable, yet a given destination cannot be reached at every sufficiently large step count. Starting at $c_1$ produces alternating one-time probability distributions. See Hao Wu's MIT lecture 2, irreducibility and periods for the definitions.

Adding a holding probability of a Markov chain $\epsilon$ at each state, with $0<\epsilon <1$, givesThe arrows between the two states remain, so the Markov chain remains irreducible. A positive one-step return makes it an aperiodic Markov chain. At $\epsilon =1/2$, both rows are $(1/2,1/2)$ and $C_{1/2}^n=C_{1/2}$ for every $n\ge 1$: the initial state is forgotten after one step. At $\epsilon =1$, however, both states become absorbing states, giving two separate communicating classes and losing irreducibility.

For a finite Markov chain, irreducibility guarantees a unique stationary distribution with positive weight at every state. Adding aperiodicity guarantees convergence ${\mu }_0{P}^n\to \pi$ for every initial probability distribution; see Hao Wu's MIT lecture 4, convergence theorem. The Markov chain ergodic theorem gives convergence of empirical visit frequencies even for an irreducible periodic Markov chain. These results explain why checking connectivity is a useful first step before studying equilibrium. Conversely, any absorbing state prevents irreducibility if there is another state: it cannot reach that state. The one-state Markov chain is the exception.

The term Markov chain commonly denotes a Markov process on a finite or countable state space. Time may be discrete, as for a daily record, or continuous, as for a continuous-time Markov chain. Some authors also use “Markov chain” for discrete-time processes on more general state spaces; specifying both time and state space avoids this ambiguity.

For a Discrete-time Markov chain, its initial probability distribution ${\mu }_0$ and transition matrix $P$ specify the probability of each finite path:This is the Markov factorization, obtained by applying conditional probability repeatedly. See Hao Wu's MIT lecture 1, definition and distribution evolution.

For the constructed sunny/rainy Markov chain below, starting sunny makes the path sunny, rainy, sunny have probability $1\times 0.2\times 0.4=0.08$. Starting instead with stationary distribution $(2/3,1/3)$ makes that path's probability $(2/3)\times 0.2\times 0.4=4/75$. The transition probabilities are identical in both experiments; the initial probability distribution distinguishes them.

A policyholder's future insurance premiums depend on a finite discount level, updated according to whether a claim is submitted in each year. With independent yearly accidents and repair costs, and a claim decision depending only on the current level and cost, the level is a Discrete-time Markov chain. Accident probability and submitted-claim probability differ when the policyholder pays small repair costs without claiming.

Record the discount level at the start of each policy year as the state. A no claims discount system is a Discrete-time Markov chain when, conditional on this state, the next year's claim behavior and update rule do not depend on earlier history. Independent annual accident and loss mechanisms with a fixed state-based reporting rule give this property; one should not equate the probability of an accident with the probability of a submitted claim.

The density identifies $Z=(\log L-\mu )/\sigma$ as a standard normal random variable: substitute $x=e^{\mu +\sigma z}$ and $dx=\sigma xdz$. Therefore, for $a>0$,Here $L$ is the numerical repair cost. The character preceding $L$ in the PDF is a currency symbol, not a separate multiplier.

Order states as $0\%,20\%,30\%$. With the stipulated common annual probability $p$ of no submitted claims, the transition matrix isA claim resets any state to zero, while a no-claim year raises the discount one level or keeps the top level fixed. If claim probabilities depend on the state, the corresponding row instead uses $p_i=\mathbb{P}(\text{no submitted claim}\mid \text{state }i)$; the same transition mechanism applies. The supplied single $p$ is a simplifying model assumption.

For a policyholder currently at zero discount, the current premium $C$ has the same value under either decision and cancels in the comparison. If the repair is self-funded and no later accidents occur, the next two premiums are $0.8C$ and $0.7C$, giving a total cost $L+2.5C$ including the current premium. If a claim is made, the next two premiums are $C$ and $0.8C$, giving total cost $2.8C$, with the repair paid by the insurer. Thus the two-year no-claims threshold with reset to zero isAt equality the holder is indifferent; it has probability zero under the continuous density. This assumes full reimbursement and no excess or deductible, as in the stated model.

Put $a=0.3C$. The conditional expected claim is $\mathbb{E}[L\mid L>a]$. Completing the square in the normal integral gives the truncated lognormal momentDividing by the reporting probability gives the expected size of a submitted claim for the full-premium holder:All monetary amounts must use the same unit despite the PDF's mixed currency notation. The other two current levels both have next-two-year no-claim premiums $0.7C+0.7C$, versus $C+0.8C$ after a claim, so their thresholds are $0.4C$. Consequently incorporating this behavioral decision into claim probabilities generally produces state-specific $p_i$, rather than deriving a common $p$ from the severity density alone; no accident-frequency law is provided.

Let $Z_m$ be the discount level during policy year $m$, with $Z_0=1$. Under the usual assumption of independent successive annual accidents and repair costs with the same laws each year, the submitted-claim decision depends only on $Z_m$ and that year's cost. Consequently the conditional distribution of $Z_{m+1}$ depends on the past only through $Z_m$: this is the Markov property. The no claims discount system is therefore a time-homogeneous Discrete-time Markov chain. Stationarity of accident probabilities alone, without this independence assumption, would not by itself establish the Markov property.

At level one a claim keeps the policyholder at one, while no submitted claim moves them to two. At level two, a submitted claim moves them to one and no claim to three. At level three, a claim moves them to two and no claim keeps them at three. Thus $p_i$ in the specified transition matrix is the annual probability of a submitted claim at current level $i$, not merely of an accident. A small accident paid privately counts as no claim for these transitions.

Let $b>0$ denote the undiscounted annual premium. It is not specified numerically in the source; costs can instead be measured in units of that premium, giving $b=1$. The actual premium payments at the three levels are $b$, $b(1-\alpha )$, and $b(1-\beta )$. Compare the two future paths, assuming no further accidents, to obtain the two-year premium-loss claim threshold. From level one, a current claim leads to future levels $(1,2)$, whereas no claim leads to $(2,3)$, giving extra premiumFrom level two the two paths are $(1,2)$ and $(3,3)$, and from level three they are $(2,3)$ and $(3,3)$. HenceThese are all strictly positive. The PDF's discount fractions are $\alpha$ and $\beta$; the converted TeX has lost these factors in the percentages.

For a repair cost $X$ with the specified lognormal distribution, $\log X\sim N(\mu ,{\sigma }^2)$. Putwhere $\Phi$ is the standard normal distribution function. Conditional on an accident, the submitted-claim probability is $\mathbb{P}(X>h_i)=\overline{\Phi }(z_i)$. Multiplying by the accident probability givesSet $b=1$ for the premium-unit convention. Without a monetary premium scale, discount percentages alone cannot determine a numerical threshold for a monetary repair cost.

This nearest-neighbour Markov chain is a finite birth-death chain. Its stationary distribution $\pi$ satisfies the endpoint balance equationsThese detailed balance equations also imply the middle-state balance equation. The three-level no claims discount equilibrium is thereforewhereSince each $p_i$ lies strictly between zero and one, this is an irreducible Markov chain. Positive holding probabilities at the endpoints also make it an aperiodic Markov chain. Its unique stationary distribution is the limiting distribution even from the initial level one, and gives the steady-state proportions for a homogeneous population of such policyholders.

A submitted claim at level one conditions the repair cost on $X>h_1$. The accident probability cancels in this conditional expectation. For $z_1=(\log h_1-\mu )/\sigma$, the truncated lognormal moment follows by completing the square:Dividing by $\mathbb{P}(X>h_1)=\overline{\Phi }(z_1)$ yields the expected size of an actual submitted claim:The unconditional expected amount paid per year from level one would instead multiply its tail first moment by $p$; it is not the conditional claim size requested here.

A stationary distribution is a probability distribution $\pi$ preserved by a Markov chain's evolution. For a Discrete-time Markov chain in the row convention,Thus $\pi$ is a normalized nonnegative left eigenvector of the transition matrix for eigenvalue one. Starting with $X_0$ distributed as $\pi$ makes every $X_n$ have that same probability distribution. For a time-homogeneous Markov chain, the path probabilities are also unchanged by shifting all observation times together: the process has strict stationarity. This does not mean its individual paths are constant. See Hao Wu's MIT lecture 2, stationary distributions.

For the sunny/rainy transition matrix, write $\pi =(a,1-a)$. The sunny component of $\pi P=\pi$ gives $a=0.8a+0.4(1-a)$, so $0.6a=0.4$ andMoreover,These equal stationary flows verify detailed balance and show that the Markov chain is reversible. The diagonal detailed balance equations hold automatically. Only $1/3$ of the stationary days are rainy, but $0.6$ of the days following a rainy day are rainy: stationarity permits dependence.

A finite Markov chain that is an irreducible Markov chain has a unique stationary distribution. If it is also an aperiodic Markov chain, ${\mu }_0{P}^n\to \pi$ for every initial probability distribution. The sunny/rainy example satisfies both conditions: both directions are possible and each state has a positive holding probability of a Markov chain. See Hao Wu's MIT lecture 4, convergence theorem. For an infinite state space, the additional positive recurrence hypothesis in the countable-state Markov chain convergence theorem matters.

Existence of a stationary distribution alone does not give this convergence. Consider the deterministic two-cycleHere $\pi C=\pi$, but starting in the first state gives alternating probability distributions $(1,0),(0,1),(1,0),\dots$. Its period of a state in a Markov chain is two. The running fraction of visits to either state nevertheless tends to $1/2$, illustrating the Markov chain ergodic theorem. Starting with $\pi$ instead keeps the one-time probability distributions stationary even though every path alternates.

A row-stochastic matrix has nonnegative entries and each row sums to one. For a time-homogeneous Discrete-time Markov chain, its transition matrix has entriesIf a state has zero probability at time $n$, this row is specified by the model's Markov kernel, rather than recovered by conditioning on that zero-probability event. Rows index departure states and columns arrival states. Writing the current probability distribution as a row vector ${\mu }_n$ givesThe row-sum condition ensures that matrix multiplication preserves total probability. Under a column convention the evolution uses the transpose; the convention must accompany the formula.

For the sunny/rainy transition matrix, direct matrix multiplication givesIn particular, $(P^2{)}_{SS}=0.8\times 0.8+0.2\times 0.4=0.72$. The two terms sum the disjoint paths $S\to S\to S$ and $S\to R\to S$. Requiring both subsequent days to be sunny has probability $0.64$; requiring only the second day to be sunny permits the extra path through rain.

Generally $(P^n{)}_{ij}$ is the $n$-step transition probability. The identity $P^{m+n}=P^m{P}^n$ is the discrete-time Chapman-Kolmogorov equation: sum over the intermediate state. This makes matrix powers useful for forecasting many steps without enumerating every path individually.

For a time-homogeneous continuous-time Markov chain, $P_{ij}(t)={\mathbb{P}}_i(X_t=j)$ and the matrices satisfy $P(s+t)=P(s)P(t)$, the Chapman-Kolmogorov equation. On a finite state space with Q-matrix $Q$, the matrix exponential gives $P(t)=e^{tQ}$.

The distinction from a discrete-time transition matrix is the elapsed-time parameter: $P(t)$ describes where the process is after $t$ time units, allowing any number of intervening jumps. Each $P(t)$ is a stochastic matrix, with $P(0)=I$. A Q-matrix instead records instantaneous rates and has row sums zero. Sampling at a fixed spacing $\Delta$ yields a Discrete-time Markov chain with transition matrix $P(\Delta )$. See Hao Wu's MIT lecture 21, transition semigroups and lecture 22, infinitesimal rates.