 Actor-critic architecture

An actor learns an action-selection policy while a critic learns a value prediction used to evaluate outcomes. A temporal-difference error can update both. With an accurate on-policy state-value function, its conditional mean given a state and action is that action’s advantage, supplying a useful policy-learning direction without a labelled correct action.