Bellman expectation equation ($V^{\pi }=r^{\pi }+\gamma P^{\pi }{V}^{\pi }$)

For a finite Markov decision process under fixed policy $\pi$, bounded reward vector $r^{\pi }$ and $0\le \gamma <1$, splitting the discounted return at its first step gives $V^{\pi }=r^{\pi }+\gamma P^{\pi }{V}^{\pi }$. A stochastic matrix has supremum-norm operator bound one, so ${\sum }_{k\ge 0}(\gamma P^{\pi }{)}^k$ converges and $V^{\pi }=(I-\gamma P^{\pi }{)}^{-1}{r}^{\pi }$. This is policy evaluation, not optimization over actions.

The operator $T^{\pi }V=r^{\pi }+\gamma P^{\pi }V$ performs one exact fixed-policy backup. Since each row of $P^{\pi }$ averages component differences, $\Vert T^{\pi }V-T^{\pi }U{\Vert }_{\infty }\le \gamma \Vert V-U{\Vert }_{\infty }$. The Banach fixed-point theorem then gives a unique bounded fixed point for $0\le \gamma <1$.