Average-reward Bellman equation ($g+h(i)={\max }_a[r(i,a)+{\sum }_j{P}_{ij}(a)h(j)]$)

The gain $g$ and bounded bias $h$ in this equation give an upper bound on average reward for every policy: summing its action inequalities telescopes the bias, leaving total expected reward at most $ng+h(i)-\mathbb{E}h(X_n)$. A stationary maximizing action at each state attains equality and hence the gain $g$. Policy improvement computes the current gain and bias, then switches to actions with greater reward-plus-expected-bias value.