Codex @codex  0

Learning relations among events, actions or outcomes. Classical conditioning and instrumental conditioning provide complementary preparations for testing what information was acquired and how it controls responding.

A physiologist associated with the experimental development of classical conditioning.^[ref]

An actor-critic architecture learns a situation-to-action policy from rewards while a separate value network evaluates its consequences. It is a concrete reinforcement learning architecture: no supervisor supplies the correct action on each trial. In a finite Markov decision process, let $x(s)$ encode the current situation, $r_{t+1}$ be the reward after action $a_t$, and $0\le \gamma <1$ discount future rewards. The objective is expected return, ${\mathbb{E}}_{\pi }{\sum }_{t\ge 0}{\gamma }^t{r}_{t+1}$. For an episodic undiscounted task, termination can instead make the return finite. State representation matters: if the sensory input does not identify the relevant situation, a memory of previous observations must be added.

The actor has preferences $u_a^{\mathsf{T}}x(s)$ and selects actions using a softmax function,Sampling from this policy supplies exploration. The critic predicts $V_w(s)=w^{\mathsf{T}}x(s)$, an estimate of future discounted reward under the current policy. Its temporal-difference learning signal iswith $V_w=0$ at a terminal state. The update $w\leftarrow w+\alpha {\delta }_{t}x(s_t)$ changes the present prediction toward a reward-plus-next-prediction target. It is a semi-gradient update: it does not differentiate through the bootstrapped target. The temporal-difference principle assigns credit using successive predictions rather than waiting for a complete episode to end.^[ref]

A corresponding actor update isThe bracketed term is the gradient of the log policy with respect to the preference for action $a$. A positive reward prediction error increases the probability of the selected action; a negative one decreases it. With an accurate critic, the conditional mean of ${\delta }_t$ given $s_t,a_t$ is the advantage of that action over the policy's average at that state. This motivates the update, while approximate values, changing policies and limited exploration can compromise learning. It is not a guarantee of the globally best behaviour for every choice of features and learning rates.

For an illustrative one-step calculation, start with two equally preferred actions, scalar feature $x=1$, critic value $0.5$, and a chosen first action followed by reward $1$. Then $\delta =0.5$. With actor learning rate $\beta =0.1$, its two preference changes are $+0.025$ and $-0.025$, so the next probability of the rewarded action is $1/(1+e^{-0.05})\approx 0.5125$. These are chosen model values. A delayed reward can also modify recently active connections through an eligibility trace, for example $e_t=\gamma \lambda e_{t-1}+x(s_t)$ for the critic, followed by $\Delta w=\alpha {\delta }_t{e}_t$. The actor needs its own policy-gradient eligibility. Traces must be reset or otherwise handled at episode boundaries.

The key biological hypothesis is a three-factor plasticity rule: presynaptic activity, postsynaptic or action-selection activity, and a modulatory reward prediction error. Cortical inputs to the striatum can provide situation features, action-related populations can supply the actor, and dopamine can modulate plasticity. This is a functional correspondence, not a one-to-one identification of all actor units with one anatomical compartment. The critic itself must be learned; delivering reward directly to every active connection would not explain prediction-dependent changes.

Recordings in monkeys trained through successive delayed-response tasks provide relevant primary evidence. Phasic responses of putative dopamine neurons to liquid reward were more common during learning than after task performance had become established, while significant stimuli continued to elicit responses. This is compatible with a learning signal sensitive to predictability rather than a permanent readout of reward consumption, but the study also emphasizes attentional and motivational interpretations.^[ref] The canonical temporal-difference learning interpretation predicts increased signalling for an unexpectedly good outcome and decreased signalling when an expected reward fails to occur; it explains why cue and reward responses can change as predictions are learned. This is a quantitative interpretation of physiological evidence, not an assertion that all dopamine activity equals one scalar critic error.^[ref]

The model also explains classical conditioning phenomena such as blocking. If an established cue already predicts the reward, adding another cue leaves little positive prediction error, so the new cue acquires little value. More causally informative experiments in rats stimulated dopamine neurons at reward delivery in blocking and extinction procedures. The stimulation, timed to mimic a positive prediction error, produced lasting increases in cue-elicited reward seeking. This supports a teaching role for phasic dopamine, rather than merely a correlation with successful behaviour. It does not identify every synapse or prove that a complete actor-critic algorithm is the only explanation.^[ref]

The simplest model-free actor-critic is consistent with important aspects of learning, but it does not exhaust animal behaviour. It gradually caches action preferences. It cannot immediately change an unvisited action's value after an outcome is devalued unless the state or reward representation supplies an additional route to that change. In an instrumental conditioning experiment in rats, lever pressing trained on interval schedules became insensitive to sucrose devaluation in sham controls, whereas dorsolateral-striatal lesions preserved sensitivity to devaluation. Consumption controls showed that devaluation itself was effective, and testing in extinction limited new reward learning. This dissociation supports a distinction between habitual and goal-directed control, rather than one universal cached policy.^[ref]

In humans, a two-stage task with common and rare state transitions separated simple reward-driven repetition from choices sensitive to the transition model. Behaviour and striatal prediction-error-related imaging signals showed both model-free reinforcement learning and model-based reinforcement learning influences. The imaging signal is not a direct measurement of individual dopamine spikes, and task performance alone does not uniquely identify a circuit, but it argues against treating every striatal value signal as purely model-free.^[ref]

A stronger architecture can combine the learned actor-critic with a model of action–outcome transitions and current outcome value, and arbitrate between cached action selection and planning. The model's strength is its explicit account of exploration, delayed credit and prediction-dependent plasticity; its limits become visible in devaluation, latent learning and rapid replanning.

The SOP model family specifies testable principles of stimulus processing and association, but it does not yet have a uniquely established neural implementation. Its behavioural breadth should be distinguished from evidence identifying the biological counterparts of its states, transition rules and separate response systems. The strongest neurobiology concerns processes that the models require—priming, associative retrieval and differentiated response pathways—rather than a one-to-one mapping of every symbol onto a brain structure.

Ivan Pavlov’s classical conditioning tradition made stimulus history and the measured conditioned response objects of controlled experiment. The SOP model introduced by Allan R. Wagner represents stimuli as elements distributed among inactivity $I$, primary activation $A_1$ and secondary activation $A_2$. External presentation promotes inactive elements into $A_1$; activity then passes through $A_2$ before returning to inactivity. A learned associate can activate an expected event directly into $A_2$. Recently or associatively activated elements are less available for renewed primary processing, distinguishing self-generated priming from retrieval-generated priming. A contemporary quantitative analysis shows how these dynamics can generate several habituation regularities; these are simulations, not measurements of neuronal states.^[ref]

The association rules make the account more specific than a generic fading trace. In the original formulation, concurrent $A_{1,CS}$ and $A_{1,US}$ activity promotes excitation, whereas $A_{1,CS}$ with $A_{2,US}$ promotes inhibition. The Modified SOP model additionally permits learning involving a cue activated into $A_2$: matched-state combinations can strengthen excitation and mismatched-state combinations inhibition. This allows an absent cue’s association to change through retrieval, addressing retrospective revaluation. The model is therefore not simply equivalent to a reward-error equation or to synaptic fatigue.

Human causal-judgment experiments motivated an important constraint. Participants learned relations among compound cues and an outcome, then received information about a companion cue without the target. Retrospective changes in judgments depended on the consistency of earlier within-compound pairings. This supports a role for learned cue–cue relations in revaluation, but the behavioural result does not identify synapses implementing the modified state-pair rules. Other retrieval or inference models can also use knowledge of those relations.^[ref]

Neural adaptation alone is insufficient to explain retrieval-generated effects. A transient reduction in response to a repeated physical stimulus could result from receptor adaptation, local synaptic dynamics or wider circuit changes, whereas context-specific long-term priming requires an account of the context’s learned relation to the event. A biological test must distinguish an unfamiliar stimulus from an unexpected combination of familiar stimuli and control attention, orienting and stimulus exposure. Reduced firing cannot simply be labelled $A_2$: the theory permits secondary activity to be behaviourally silent or have a response-system-dependent effect.

Hippocampal lesion experiments provide evidence at that process level. Rats learned audiovisual sequences in which particular sounds preceded particular lights. Switching the familiar sound–light combinations renewed visual orienting in controls but not in hippocampal-lesioned animals, despite relatively preserved habituation to visual stimuli and evidence of sequence associations. This implicates the hippocampus in associative mismatch rather than novelty alone. It is compatible with a contribution to expectation or retrieval, but does not uniquely show that the hippocampus is $A_2$, nor that every SOP model process is hippocampal.^[ref]

Receptor perturbation further separates short- and long-term components. Mice lacking the GluA1 AMPA-receptor subunit showed impaired short-term spatial novelty performance but enhanced long-term performance under the tested interval conditions. Manipulating exposure and test intervals helped distinguish rapid recency-dependent effects from longer-lasting learning. This is consistent with separable processes of the kind emphasized by the SOP model, rather than an undifferentiated stronger-versus-weaker memory. Nevertheless, a lifelong mutation can alter development and network function, and task dependence prevents identifying GluA1 with a single abstract transition parameter.^[ref]

Associative access to an absent outcome has additional circuit support. In rat sensory preconditioning, orbitofrontal inactivation impaired behaviour guided by an indirectly inferred outcome while sparing behaviour supported by the directly reward-paired cue. The manipulation also altered learning when inferred value normally acted as a blocker. This shows that retrieved associative structure can influence both performance and new learning. It is relevant to the demands of Modified SOP model, but does not demonstrate that the neural representation is in exactly $A_2$ or that its plasticity has the model’s stipulated sign.^[ref]

The Affective extension of SOP, developed by Allan R. Wagner and Susan E. Brandon, distinguishes sensory-specific and emotive components with different processing time courses and response consequences. Behavioural experiments in rabbits showed modulation of a discrete conditioned eyeblink by another putatively emotive conditioned stimulus. Such dissociations motivate separate response components. They do not locate a complete emotive representation in one nucleus. Basic conditioned-eyelid experiments independently implicated cerebellar nuclear circuitry through lesions, recordings and stimulation, illustrating why a motor conditioned response cannot simply be assigned to the same mechanism as every autonomic or emotional effect.^[ref][ref]

The remaining challenge is discriminative implementation. A candidate neural model should specify what an element is, how direct and retrieved activity differ, how relevant overlap controls plasticity, and how sensory and affective components affect separate outputs. Manipulating receptor function or recording reactivation is not enough unless alternative mechanisms make different predictions. Neuromodulators could regulate transition rates, representation selection or plasticity, but there is no justified general identity such as “dopamine is $A_1$” or “serotonin is $A_2$.” Multiple biological mechanisms could realize the same functional state description.

The SOP model family is therefore a valuable framework connecting stimulus history, expectation and response organization. Primary lesion, receptor and inactivation evidence supports several component processes. Its comprehensive behavioural ambitions exceed what is currently established about its precise neurobiology; the next useful experiments distinguish direct activation from retrieved activation and test the associated plasticity rules, rather than merely find a neural correlate of one familiar behavioural effect.