Markov reward model

A Markov reward model assigns a reward or cost rate $r_i$ to each state of a Continuous-time Markov chain and accumulates it with elapsed time:For a finite chain with generator $Q$ and reward column vector $r$, its expected accumulated reward isIntegrating state indicators proves the expectation formula; differentiating the matrix exponential gives the differential equation. Indicator rewards count time in a selected state. If rates represent power consumption, the accumulated reward is energy.

For a two-state chain $0\to 1$ at rate $a>0$ and $1\to 0$ at rate $b>0$, let only state one earn unit reward. Starting at zero,It is the expected duration in state one, obtained by integrating its occupancy probability. With exponential discount $\delta >0$, the infinite-horizon expected reward instead is $(\delta I-Q{)}^{-1}r$, the resolvent applied to $r$. Bounded rewards and positive discount make that integral finite even when the undiscounted cumulative reward grows indefinitely.