= Tabular temporal-difference policy evaluation
{title2=$V(S_t)\leftarrow V(S_t)+\alpha_t\delta_t$}
A sampled transition updates only the visited value: $V(S_t)\leftarrow V(S_t)+\alpha_t\delta_t$, where $\delta_t=R_{t+1}+\gamma V(S_{t+1})-V(S_t)$. Under the fixed policy, its conditional mean is $(T^\pi V)(s)-V(s)$. Standard tabular convergence requires adequate state visitation and step-size/noise conditions; this mean identity alone does not prove convergence for arbitrary function approximation.
Back to article page
