Learning to choose actions from their consequences in an environment, with the objective of increasing expected reward or return. Reward does not normally specify a correct action label. State representation, exploration and temporal credit assignment are central computational requirements.
Learning about a distribution of possible future returns rather than only its mean. Multiple prediction channels can vary in optimism and sensitivity to positive and negative errors. Such population heterogeneity can refine a reward prediction error account without implying that every dopamine signal performs the same computation.
Using stored or regenerated experience for further learning, often interleaved with new observations. It can improve data reuse and retention, but its sampling distribution and fidelity affect what objective is learned. A software replay buffer is not a complete biological model of hippocampal reactivation.
Learning or using a model of transitions and outcomes to evaluate possible actions. It can support flexible replanning when the value of an outcome changes, but depends on the learned model and computational resources. Behavioural experiments must distinguish it from alternative strategies with similar observable predictions.
Learning cached values or action preferences without explicitly planning through a model of state transitions and outcomes. A basic actor-critic is an example. The absence of a transition model can limit rapid revaluation of an unvisited action after its outcome changes.
A value prediction is updated using a reward and a subsequent prediction: . The linear semi-gradient rule is for ; the target is held fixed for this update. Terminal-state value is zero in an episodic task.[ref]
Learning to predict by the methods of temporal differences by Richard S. Sutton. Abstract and sections 2–3; Machine Learning 3, pp. 9–44: successive predictions and temporal credit assignment
A sampled transition updates only the visited value: , where . Under the fixed policy, its conditional mean is . Standard tabular convergence requires adequate state visitation and step-size/noise conditions; this mean identity alone does not prove convergence for arbitrary function approximation.
A discrepancy between obtained or newly predicted reward and the reward previously expected. In a temporal-difference model it includes the next-state prediction, rather than being simply the immediate reward. A positive value signals a better-than-predicted outcome; a negative value signals a worse-than-predicted one.
An actor learns an action-selection policy while a critic learns a value prediction used to evaluate outcomes. A temporal-difference error can update both. With an accurate on-policy state-value function, its conditional mean given a state and action is that action’s advantage, supplying a useful policy-learning direction without a labelled correct action.
