A value prediction is updated using a reward and a subsequent prediction: . The linear semi-gradient rule is for ; the target is held fixed for this update. Terminal-state value is zero in an episodic task.[ref]
Learning to predict by the methods of temporal differences by Richard S. Sutton. Abstract and sections 2–3; Machine Learning 3, pp. 9–44: successive predictions and temporal credit assignment
A sampled transition updates only the visited value: , where . Under the fixed policy, its conditional mean is . Standard tabular convergence requires adequate state visitation and step-size/noise conditions; this mean identity alone does not prove convergence for arbitrary function approximation.
A discrepancy between obtained or newly predicted reward and the reward previously expected. In a temporal-difference model it includes the next-state prediction, rather than being simply the immediate reward. A positive value signals a better-than-predicted outcome; a negative value signals a worse-than-predicted one.
Articles by others on the same topic
Temporal Difference (TD) learning is a central concept in the field of reinforcement learning (RL), which is a type of machine learning concerned with how agents ought to take actions in an environment in order to maximize some notion of cumulative reward. TD learning combines ideas from Monte Carlo methods and Dynamic Programming. Here are some key features of Temporal Difference learning: 1. **Learning from Experience:** TD learning allows an agent to learn directly from episodes of experience without needing a model of the environment.