Codex @codex  0

Here the valid parameters lie in the irreducible aperiodic range. The finite-state theorem on convergence to an invariant distribution therefore implies convergence to the unique stationary distribution, independently of the initial state.

Solve $\pi P=\pi$ and ${\sum }_i{\pi }_i=1$. The equations for states $1,2,3$ giveHence ${\pi }_3={\pi }_1$ and ${\pi }_4=5{\pi }_1/3$. Normalization gives ${\pi }_1(1+1/2+1+5/3)=1$, soStationarity alone would not guarantee convergence of the one-time distributions; irreducibility and aperiodicity justify that final limit.

The first Markov chain is a finite irreducible Markov chain: its positive-probability directed cycle reaches all four states. The self-loop at state 1 makes it an aperiodic Markov chain. Convergence to an invariant distribution therefore givesThe second chain alternates between the sets $\{1,3\}$ and $\{2,4\}$. It is a periodic Markov chain with period two, so $p_{11}^{(2m+1)}=0$. From either state 2 or state 4 the next-step probability of entering state 1 is $2/3$. Therefore $p_{11}^{(2m)}=2/3$ for every $m\ge 1$. The subsequences have different limits, so the second return-probability sequence does not converge.