= Discrete-time Markov chain
{c}
{title2=$\mathbb P(Y_{n+1}=j\mid\mathcal F_n)=P_{Y_nj}$}
A time-homogeneous <Discrete-time Markov chain> is a sequence of <random variables> $Y_0,Y_1,\ldots$ indexed by nonnegative <integers>, with the displayed <conditional probabilities>. Here $\mathcal F_n$ is its <natural filtration> and $P$ is a row-stochastic <transition matrix>. The <Markov property> removes dependence on earlier observations once $Y_n$ is specified; the <time-homogeneous Markov property> makes $P$ independent of $n$.
Consider states $S$ (sunny) and $R$ (rainy), with the constructed <transition matrix>
$$
P=\begin{pmatrix}0.8&0.2\\0.4&0.6\end{pmatrix}.
$$
Rows give today's state and columns tomorrow's, in order $S,R$. These numbers define an illustrative model, with one step interpreted as one day; they are not estimates of real weather. Let $q_n=\mathbb P(Y_n=S)$. The <law of total probability> gives the <recurrence relation>
$$
q_{n+1}=0.8q_n+0.4(1-q_n)=0.4+0.4q_n.
$$
Subtracting $2/3$ makes the deviation multiply by $0.4$ at each step. Thus
$$
q_n=\frac23+\left(q_0-\frac23\right)0.4^n.
$$
Starting sunny gives $q_0=1$, $q_1=0.8$, $q_2=0.72$ and $q_3=0.688$; starting rainy gives $q_0=0$ and $q_2=0.56$. The two predictions approach the same <stationary distribution> while retaining a shrinking trace of the initial <probability distribution>.
\Image[/markov-chain-convergence.png]
{title=Constructed sunny/rainy transitions and the exact sunny probabilities from three initial distributions.}
An <expected hitting time> answers a different question. Starting sunny, let $T_R=\inf\{n\geq0:Y_n=R\}$. Every day before the first rain remains sunny, so
$$
\mathbb P_S(T_R=k)=0.8^{k-1}0.2\quad(k\geq1).
$$
This <geometric distribution> has finite <expected value>. A first-step calculation gives $h_S=1+0.8h_S+0.2h_R$, with $h_R=0$, hence $h_S=5$ days. The same result follows from the <geometric series> $\mathbb E_ST_R=\sum_{n\geq0}0.8^n=5$. This is a mean waiting time, not a prediction that rain first occurs exactly five days later.
A <probability> row <vector> $\pi$ is a <stationary distribution> when $\pi P=\pi$. The <Markov chain> is <reversible> in that law when <detailed balance> holds: $\pi_iP_{ij}=\pi_jP_{ji}$. Multiplying those equations along a finite path gives equal forward and reversed stationary path <probabilities>.
Back to article page
