= Average-reward Bellman equation
{title2=$g+h(i)=\max_a[r(i,a)+\sum_jP_{ij}(a)h(j)]$}
The gain $g$ and bounded bias $h$ in this equation give an upper bound on average reward for every policy: summing its action inequalities telescopes the bias, leaving total expected reward at most $ng+h(i)-\mathbb Eh(X_n)$. A stationary maximizing action at each state attains equality and hence the gain $g$. Policy improvement computes the current gain and bias, then switches to actions with greater reward-plus-expected-bias value.
Back to article page
