= Markov reward model
{c}
{wiki}
A <Markov reward model> assigns a reward or cost rate $r_i$ to each state of a <Continuous-time Markov chain> and accumulates it with elapsed time:
$$
A_T=\int_0^T r_{X_t}\,dt.
$$
For a finite chain with generator $Q$ and reward column <vector> $r$, its expected accumulated reward is
$$
v(T)=\int_0^T e^{tQ}r\,dt,\qquad v'(T)=Qv(T)+r,\qquad v(0)=0.
$$
Integrating state indicators proves the <expected value>[expectation] formula; differentiating the <matrix exponential> gives the differential equation. Indicator rewards count time in a selected state. If rates represent <power>[power consumption], the accumulated reward is <energy>.
For a two-state chain $0\to1$ at rate $a>0$ and $1\to0$ at rate $b>0$, let only state one earn unit reward. Starting at zero,
$$
v_0(T)=\frac a{a+b}\left[T-\frac{1-e^{-(a+b)T}}{a+b}\right].
$$
It is the expected duration in state one, obtained by integrating its occupancy <probability>. With exponential discount $\delta>0$, the infinite-horizon expected reward instead is $(\delta I-Q)^{-1}r$, the <resolvent operator>[resolvent] applied to $r$. Bounded rewards and positive discount make that <integral> finite even when the undiscounted cumulative reward grows indefinitely.
Back to article page
