Tabular temporal-difference policy evaluation ($V(S_t)\leftarrow V(S_t)+{\alpha }_t{\delta }_t$)

A sampled transition updates only the visited value: $V(S_t)\leftarrow V(S_t)+{\alpha }_t{\delta }_t$, where ${\delta }_t=R_{t+1}+\gamma V(S_{t+1})-V(S_t)$. Under the fixed policy, its conditional mean is $(T^{\pi }V)(s)-V(s)$. Standard tabular convergence requires adequate state visitation and step-size/noise conditions; this mean identity alone does not prove convergence for arbitrary function approximation.