 Solution

The payoff table is blank in the original PDF, so denote its four row-player payoffs by $R,S,T,P$ for $CC,CD,DC,DD$, with the prisoner's dilemma ordering $T>R>P>S$. No numerical entries are inferred. Let the two reactive strategies be $(p_1,q_1)$ and $(p_2,q_2)$, where $p_i$ is cooperation after the opponent's cooperation and $q_i$ cooperation after defection.

The joint action states form a four-state Markov chain. In state order $CC,CD,DC,DD$, its transition matrix isThe prescribed initial memories give independent first actions with cooperation probabilities $p_1,p_2$. Same-round actions remain independent: each player's next action uses the other player's previous action and its own independent random draw. Thus their marginal cooperation probabilities obeyPut $a_i=p_i-q_i$. If $|a_1{a}_2|<1$, the two-step recurrences contract and the long-run payoff of reactive strategies follows fromThe joint stationary distribution is $(x_{\ast }{y}_{\ast },x_{\ast }(1-y_{\ast }),(1-x_{\ast })y_{\ast },(1-x_{\ast })(1-y_{\ast }))$. Player 1's mean payoff is $R{x}_{\ast }{y}_{\ast }+S{x}_{\ast }(1-y_{\ast })+T(1-x_{\ast })y_{\ast }+P(1-x_{\ast })(1-y_{\ast })$; exchange $S,T$ for player 2.

The deterministic boundaries must retain the initial memories rather than divide by zero. Two Tit for tat players cooperate forever and earn $R$. Two players who always do the opposite of the opponent's previous action alternate $DD,CC$ and average $(P+R)/2$. One of each cycles through all four states and each averages $(R+S+T+P)/4$. These are the only cases with $|a_1{a}_2|=1$. A periodic Markov chain need not have convergent state probabilities, but its long-time average payoff is defined by its cycle frequencies.