Computational neuroscience course of the University of Oxford

Part II advanced option in the held 2022 papers. The source examination is shared by the psychology (experimental) course of the University of Oxford and the psychology, philosophy and linguistics course of the University of Oxford.

This undergraduate Part II paper is treated in full: all six questions are solved despite the instruction to select three. The accounts distinguish proposed computations, simulation results and physiological evidence.

Public source: Released examination-paper archive, member a10967w1.pdf.

The cover identifies the undergraduate EP/PPL Second Public Examination on 26 May 2022.^[ref]

A two-layer heteroassociative memory can retrieve one activity pattern from a different associated cue through learned feedforward synapses. Let $x^{\mu }\in {\mathbb{R}}^n$ represent a cue and $t^{\mu }\in {\mathbb{R}}^m$ its associate. An input population projects to an output population through $W$, with retrieval $y=\varphi (Wx-\theta )$. The output activation function can be linear for an analysis or thresholded with synaptic competition for a sparse categorical response. This is pattern association: the output need not reproduce the input, unlike an autoassociative memory.

During storage, the cue and its desired associate must be coactive. A local Hebbian learning rule, $\Delta W_{ij}=\eta t_i{x}_j$, strengthens the connection from an active cue neuron to an active target neuron. With normalized inputs, the idealized batch weights arePresenting cue $x^{\nu }$ givesThe first term is the desired association; the second is interference from overlapping cues. Thus retrieval follows exactly for orthonormal cue patterns, while sparse coding, centring, synaptic normalization and a competitive output can reduce interference for realistic overlapping inputs. These devices have distinct roles: synaptic normalization prevents unlimited weight growth, and synaptic competition separates competing associates. Signed, centred model weights should not be identified literally with negative excitatory synapses; biological inhibition and baseline subtraction need an explicit circuit implementation.

As an exact illustrative calculation, take $x^1=(1,0{)}^{\mathsf{T}}$, $x^2=(0,1{)}^{\mathsf{T}}$, $t^1=(1,1,0{)}^{\mathsf{T}}$ and $t^2=(0,0,1{)}^{\mathsf{T}}$. ThenA cue $(0.8,0.1{)}^{\mathsf{T}}$ produces $(0.8,0.8,0.1{)}^{\mathsf{T}}$, so threshold $0.5$ retrieves the first associate. These are chosen model values, not recorded firing rates. If two genuinely different associates share an identical cue, a deterministic feedforward map cannot choose between them without additional context; adding capacity alone does not solve that ambiguity. A cue-to-associate map also need not provide the reverse association unless the reverse pathway is separately learned.

The hippocampus provides a plausible anatomical setting. In a hippocampal episodic memory model, a retrieved hippocampal CA3 event pattern can project through the hippocampal CA1 output pathway to reinstate associated cortical content. The feedforward CA3-to-CA1 and hippocampal-to-cortical links can provide heteroassociation, while recurrent CA3 connectivity provides autoassociation. The entorhinal cortex supplies converging information and routes cortical communication. This is a computational assignment to pathways; anatomical connectivity alone does not prove that their weights implement the displayed rule.

An especially direct physiological test comes from trained visual pairs. Monkeys learned arbitrary associations between computer-generated pictures and then selected the associated picture after a delay. Extracellular single-unit recording in anterior temporal cortex found both neurons responding to the two members of a learned pair and neurons whose delay activity anticipated the appropriate associate. The preparation therefore links an experimentally controlled cue–associate relation to neural responses, rather than merely showing visual selectivity. However, the recordings do not identify the individual modified synapses, establish one-way connectivity, or exclude recurrent retrieval. They support an associative representation in temporal cortex, not a unique proof of a two-layer implementation.^[ref]

The rapid-learning requirement is relevant to the hippocampal implementation: CA3-restricted NMDA receptor deletion impaired learning of a newly located hidden platform in a delayed matching-to-place task while sparing performance with previously experienced locations. This supports rapid hippocampal storage, but does not by itself identify a specific feedforward association matrix.^[ref]

A decisive model test would measure cue and associate representations before learning, after learning, and during partial-cue retrieval, while selectively disrupting the proposed connecting pathway. Correct associate decoding with unrelated-cue and familiar-visual-response controls would be stronger evidence than activity during successful recall alone. The architecture is a plausible component of temporal and hippocampal memory circuits; its local learning rule, interference properties and pathway-specific causal predictions make it testable.

I would combine visually calibrated continuous attractor networks with competitive learning, while keeping heading, self-location and the viewed location as distinct variables. A head direction cell reports orientation, a place cell reports self-location, and a spatial view cell reports a viewed part of the external world. A model that predicts all three from the same retinal image without controlling eye position and body position risks explaining a sampling correlation rather than the neural code.

For heading, let $r_i$ be rates in a ring indexed by preferred directions ${\theta }_i$. A concrete rate model isChoose local excitation and broad inhibition strengths so that one localized activity bump is stable; a bump is not guaranteed for arbitrary connection strengths. An asymmetric component driven by angular velocity shifts the bump, implementing path integration. An approximate decoded angle is $\hat{\theta }=\arg {\sum }_i{r}_i{e}^{\mathrm{i}{\theta }_i}$. Its periodic representation avoids an artificial discontinuity between $359^{\circ }$ and $0^{\circ }$. During active turns, correlated landmark features and the currently active direction population modify visual afferents by bounded Hebbian learning. Repeated encounters calibrate visual inputs to the internal angular state. In the absence of external cues, recurrent activity can persist while velocity noise produces drift; the landmarks correct that drift rather than having to generate every moment of activity.

This separates a model assumption from a developmental explanation. Prespecifying a ring with a moving bump assumes part of the organization; it does not show that vision constructs the ring. A stronger developmental model should start from the claimed immature connectivity and demonstrate emergence of stable tuning under realistic self-motion. The observed head direction cell system is constrained by cue-card experiments: rats explored a cylindrical enclosure, and rotating a prominent wall card rotated preferred firing directions with much less change in tuning shape. Card removal could change the angular anchor without eliminating directional firing. These results motivate learned landmark anchoring plus internally maintained heading, rather than a purely retinal feature detector.^[ref]

For place cells, use converging visual-landmark features, boundary geometry and a self-motion estimate of position. A winner-take-all network can learn prototypes of this conjunctive input. For normalized input $z$ and winning unit $k$, a simple competitive learning rule is $\Delta w_k=\eta (z-w_k)$ followed by any required synaptic normalization. Similar conjunctions recur around a location, whereas their component retinal appearances vary with heading. Multiple headings and eye positions must therefore be sampled at each location. Coupling these learned visual responses to a two-dimensional continuous attractor network or another self-motion state estimator lets vision calibrate a position representation that remains usable during brief cue loss. One must not silently supply correct world-position labels to an allegedly unsupervised developmental system: simulated position is a validation variable, and any internal position estimate needs a specified sensory origin.

For spatial view cells, the target variable is different. In a planar illustration, if the animal is at $p$, head angle is ${\theta }_h$ and eye angle relative to the head is ${\theta }_e$, a viewed point lies onA depth estimate or intersection with an inferred scene surface is needed to determine $\lambda$. Combining visual features, eye-position signals, heading and location can let competitive learning associate multiple views of the same world location. The formula specifies the geometric computation to be learned; it is not evidence that a neuron literally performs trigonometry. For example, facing east with a $30^{\circ }$ leftward gaze and facing $30^{\circ }$ north of east with straight-ahead gaze can look toward the same point from the same position, despite different head directions. This distinction is experimentally important: recordings in freely walking macaques, with eye-position measurements, found hippocampal responses related to looked-at locations rather than simply to the monkey's place or head direction.^[ref]

The main outstanding challenges are developmental causation, ambiguous sensory input and generalization across conditions. First, spatial circuitry need not wait for patterned vision: recordings before and after eye opening show an early directional signal, with immature stability. Therefore vision can calibrate or stabilize a partially organized circuit, and a theory claiming wholly visual construction must account for that developmental timing.^[ref] Second, visually similar corridors can correspond to different positions, while one position can produce very different images. Cue conflict, darkness, passive transport and novel environments separate landmark anchoring, self-motion updating and hippocampal remapping. Third, retinal direction is not world direction: eye movements, head pitch, depth, occlusion and independently moving objects challenge an allocentric spatial representation. Finally, the model must explain species and task differences without assuming that rodent place cells and primate spatial view cells exhaust all spatial responses.

I would assess held-out positions, headings and gaze combinations, compare predicted tuning and drift against recordings, and ablate the visual-learning and self-motion components separately. A successful model should predict how tuning changes after specific experience and cue manipulations, not merely fit a familiar trajectory. Visually guided development is therefore a multisensory learning problem with an experimentally constrained internal spatial dynamics.

A useful hippocampal episodic memory model assigns complementary computations to competitive learning, recurrent autoassociation and feedforward pattern association. The event must bind its participants, location and context, so that a partial cue later reinstates the appropriate conjunction. Merely classifying inputs or remembering a location is insufficient to explain an episodic memory.

A concrete pathway begins with distributed cortical input through the entorhinal cortex. The dentate gyrus expands and sparsifies this input: strong synaptic competition means that only a small fraction of granule cells respond strongly. A winning unit or winning sparse population strengthens its active afferents through constrained Hebbian learning. Similar episodes can then recruit less-overlapping outputs, providing pattern separation before storage. Separating two nearly identical episodes is useful because a recurrent memory otherwise tends to merge their cues. The desired separation must preserve relevant conjunctions; indiscriminately randomizing the input would make later retrieval impossible.

The sparse code recruits a hippocampal CA3 ensemble. Rapid, associatively induced long-term potentiation is a candidate substrate for storing the conjunction, with its induction conditions and persistence requiring experimental constraints. Its recurrent collateral synapses allow coactive elements of the event to become associated through Hebbian learning. One idealized centred storage rule iswhere $a$ is mean coding activity and ${\xi }^{\mu }$ is an event pattern. With inhibitory control maintaining sparsity, recurrent retrieval iterates $r\leftarrow \varphi (Wr+c-\theta )$ from cue $c$. A sufficiently informative partial cue can move activity into the stored event's basin of attraction; recurrent reinstatement supplies missing components. This is pattern completion. A Hopfield network makes the principle precise in a symmetric idealization with an energy landscape, but biological CA3 connectivity need not obey the exact symmetry or update assumptions of asynchronous Hopfield energy descent. Interference, spurious states and an incorrect initial cue can all produce erroneous completion.

The hippocampal CA1 and cortical return pathways provide heteroassociative memory. During the event, CA3 activity ${\xi }^{\mu }$ is paired with a cortical or CA1 target $z^{\mu }$, giving an illustrative mapping $A={\sum }_{\mu }{z}^{\mu }({\xi }^{\mu }{)}^{\mathsf{T}}$. When CA3 completes the event, $A\hat{\xi }$ reinstates its associated output representation. Thus competitive learning distinguishes inputs, autoassociation restores a stored conjunction, and pattern association translates the retrieved conjunction into content usable elsewhere. These are separable operations even if a real circuit performs them concurrently. CA1 also receives direct entorhinal information, so it cannot be treated as a passive relay without testing the contribution of that second input.

There is mechanistically relevant evidence beyond anatomical resemblance. In adult mice, CA3-restricted NMDA receptor deletion spared ordinary acquisition and retrieval of water-maze reference memory with the full cue set, but impaired retrieval when only some cues remained. CA1 place-related activity also deteriorated under partial-cue conditions. Full-cue performance is an important control against interpreting the deficit as a general inability to swim, see cues or express a learned response. This fits the prediction that recurrent associative plasticity is particularly important for pattern completion. Nevertheless, the deletion affects receptor-dependent processing throughout the targeted CA3 cells, and does not isolate only recurrent collateral plasticity or the exact storage rule.^[ref]

A separate delayed matching-to-place experiment found impaired rapid acquisition of novel platform locations after CA3-restricted NMDA receptor deletion, with previously experienced locations relatively spared. That extends the interpretation from partial-cue recall to rapid storage of new experiences; it also cautions against assigning CA3 an exclusively retrieval-related role.^[ref]

Simultaneous dentate gyrus and hippocampal CA3 recordings offer a more direct input–output comparison. In behaving rats, conflict between local and global spatial reference frames disrupted dentate population patterns, while CA3 maintained a more coherent representation related to the stored map. Comparing the two populations under the same manipulation supports the complementary pattern separation and pattern completion computations more specifically than finding different firing rates in different areas. It still does not prove that a particular recalled human episode corresponds to one fixed CA3 attractor.^[ref]

The evidence supports the computational division of labour, but the simplest implementation is incomplete. The same region can separate strongly changed inputs and complete moderately degraded ones; separation and completion are operating regimes, not immutable labels for every trial. Fast storage must coexist with protection of old memories, and recurrent completion must not overwrite a new but similar event. Novelty, inhibition and synaptic plasticity regulation are therefore functional requirements, rather than arbitrary additions. Experimental interventions can affect acquisition, retrieval, exploration and sensory input differently, so their timing and behavioural controls matter.

An unordered fixed-point memory also omits the temporal structure of an episode. The same people in the same place at different times need contextual or sequential representations; an attractor for a bag of features does not distinguish reversed event order. Rehearsal and longer-term cortical learning may change the dependence on the hippocampus. The theory should consequently predict partial-cue success, lure errors, interference and pathway-specific deficits across time, while acknowledging that neither a spatial task nor a correlation in population activity establishes the whole of episodic memory. The combined architecture supplies an explicit storage-and-retrieval mechanism, with experimentally supported components and clear remaining requirements.

Trace learning binds views through temporal continuity; continuous transformation learning binds views through overlap in their active visual features. Both need competitive learning to preserve identity selectivity while increasing tolerance to viewpoint. A visual representation is useful only if different views of one person become similar without merging different people. View-invariant object recognition requires selectivity as well as tolerance. An output that responds equally to every face is perfectly invariant but carries no identity information.

Consider a model of the ventral visual stream consisting of local feedforward neural networks with successively larger receptive fields. Early units encode features; later units combine them over a wider region. Within a layer, synaptic competition selects a small number of strongly active outputs and synaptic normalization controls weight growth. Let $x_j(t)$ be current input activity and $y_i(t)$ the competitive output. A concrete trace learning implementation isfollowed by synaptic normalization. Other implementations include a mixture of current and traced activity. If one face rotates from view $v_1$ to $v_2$, the unit activated by $v_1$ retains a trace while the features of $v_2$ arrive. Those features strengthen onto the same unit. Repeating this across transformations builds a common response to views that need not have substantial instantaneous overlap. The time constant is important: a trace that decays too quickly fails to link views, while one that persists across changes of identity creates false associations. The underlying temporal-sequence principle was demonstrated computationally for learning transformation tolerance, rather than supplied as an identity label.^[ref]

Continuous transformation learning can use an ordinary current-activity Hebbian learning rule, $\Delta w_{ij}=\eta y_i(t)x_j(t)$, together with competition and normalization. Suppose adjacent views activate overlapping input features. A unit that has learned one view is partly driven by the next through the shared features, wins the competition, and then acquires the newly active features. Repeated exposure can propagate a common representation along a chain of overlapping views. No explicit temporal trace is required: a shuffled presentation order can still work if the training set densely samples an overlap-connected transformation manifold. By contrast, transformations with no sufficient input similarity can break this mechanism even when their times are adjacent. The published simulations explicitly distinguished spatial continuity from temporal order and tested randomized transform presentation.^[ref]

A small constructed example makes the distinction concrete. Let three views have active feature sets $\{1,2\}$, $\{2,3\}$ and $\{3,4\}$. Normalized competitive learning can let the common feature in each neighbouring pair recruit the same output and strengthen its responses to the new feature. The first and third views need not overlap directly. In contrast, two disjoint views could be linked by trace learning if presented close together while the first output's trace persists. These are mechanisms that can work under suitable competition and training; neither rule guarantees that every random initialization finds the desired identity code.

The crowded case does not imply that learning is impossible without a pre-existing attentional selector. Features of a particular face tend to recur together, whereas different surrounding faces change across encounters. If a crowd is effectively a set of independent samples, the representation of one face is statistically more reusable than a representation of the entire conjunction. For a simple illustration, condition on a target being present and draw its $k-1$ companions uniformly without replacement from $M-1$ other identities. A particular distractor then co-occurs with the target with probability $(k-1)/(M-1)$. The target's own visible features recur much more consistently. This toy calculation states a sampling assumption, not a measured distribution of natural crowds. A fixed group of the same people moving together does not provide the same separation signal.

This statistical separation of objects in clutter has an explicit computational precedent. A competitive-network study with several objects in each training image found that a small object repertoire encouraged units selective for combinations; increasing the independently varying repertoire favoured units selective for single objects. Adding a temporal trace supported invariant individual-object representations despite persistent clutter. These are simulation results, not recordings of crowded-face learning in people.^[ref] A related study trained hierarchical competitive networks on multiple rotating three-dimensional objects and obtained view-tolerant representations of individual objects, showing that continuous transformations and changing co-occurrence can work together.^[ref]

In a real crowd, both mechanisms can nevertheless fail. Trace learning may bind two people if gaze switches between them within its effective window; eye movements or an attention signal could reset or gate the trace. Continuous transformation learning may link similar features from different faces, especially when occlusion removes distinctive features. Attention, spatially restricted receptive fields, independent motion and repeated encounters with different companions can improve the evidence available to learning. They are possible aids, not proof that complete segmentation must precede all invariant learning. Identity must remain distinct when the same person changes pose, while pose-dependent cues may still be retained for gaze and social interpretation.

Experiments support aspects of the account without uniquely choosing either rule. Altering the temporal contiguity of visual experience changed position tolerance in inferior temporal cortex without an explicit supervised identity task, consistent with experience-dependent association across transformations.^[ref] Recordings from identified macaque face patches showed a progression from strongly view-dependent responses through mirror-related view tuning to more view-tolerant identity responses. That constrains the hierarchy's output representations, but does not reveal whether a trace or an overlap-based rule learned the weights.^[ref]

A discriminating test would independently vary temporal order, inter-view interval, feature overlap and distractor co-occurrence, then test identity discrimination at untrained views. Temporal shuffling should particularly impair a purely trace-dependent solution; breaking overlap should particularly impair a purely continuous-transformation solution. Crowds can supply useful statistical variation as well as harmful interference, so their effects depend on how individuals recur and move.

Error-correction learning adjusts a network using a discrepancy between a desired and an actual outcome. Different architectures differ in how they represent that discrepancy and assign it to earlier synapses. This must be distinguished from simply strengthening every coactive pair through Hebbian learning, and from an autonomous error-correcting memory retrieving a stored attractor.

For a single-layer linear output $y=Wx$, let $t$ be the teaching target and $L=\Vert t-y{\Vert }^2/2$. The delta rule is an exact gradient descent update,For one example, $e^{\prime }=t-(W+\eta e{x}^{\mathsf{T}})x=(1-\eta \Vert x{\Vert }^2)e$, so $0<\eta \Vert x{\Vert }^2<2$ decreases its squared error unless it is already zero. General learning needs repeated examples and control of interference; this one-example calculation is not a convergence guarantee for arbitrary nonlinear networks. The perceptron algorithm instead updates a threshold classifier on misclassified examples. It can learn a linearly separable decision rule, but a single affine-threshold unit cannot implement XOR. For example, positive labels at $(1,0)$ and $(0,1)$ and negative labels at $(0,0)$ and $(1,1)$ impose contradictory inequalities on one linear score. More layers are needed to learn a suitable feature representation.

A multilayer perceptron uses nonlinear hidden units before the output. For squared error, write $h=\sigma (Ux)$ and $y=\sigma (Vh)$. Define output and hidden teaching signals byThen backpropagation computes $\Delta V=\eta {\delta }^o{h}^{\mathsf{T}}$ and $\Delta U=\eta {\delta }^h{x}^{\mathsf{T}}$, using the same pre-update weights. The chain rule makes these exact negative gradients. Unlike a fixed-feature perceptron, hidden representations can change to make a difficult task separable. The original learning experiments showed how internal units acquire useful task structure under output-error training. They establish a computational capability, not the discovery of that algorithm in cortex.^[ref]

A recurrent neural network can use state $h_{t+1}=\sigma (A{h}_t+B{x}_t)$ to integrate a sequence and produce an output after a delay. Error correction must now assign responsibility to earlier states as well as earlier layers. Applying the chain rule through the unrolled recurrence solves the algorithmic problem, but storing the required history and transporting a late teaching signal to an earlier synapse is biologically demanding. Repeated multiplication by the recurrent Jacobian matrix can also create the vanishing gradient problem or the exploding gradient problem. A local eligibility trace can retain a record of recent synaptic participation for modulation by a later error signal, although that is not generally an exact replacement for the full recurrent gradient.

A biologically motivated comparison is the cerebellum. A broad input representation can be expanded by granule cells; parallel fibers carry the resulting activity to Purkinje cells, while a climbing fiber provides a separate instructive input. Such a circuit resembles a supervised adaptive filter: an active input is modified in a manner depending on a teaching signal, with Purkinje output affecting downstream motor control. The correspondence is a circuit hypothesis, not a claim that the climbing-fiber event equals the signed scalar $t-y$ in every task.

An instructive experiment recorded Purkinje cells in monkeys tracking a moving visual target. A change in target direction supplied a learning instruction, and subsequent eye velocity and simple-spike activity measured trial-over-trial adaptation. Longer complex spikes were associated with larger subsequent simple-spike depression and stronger behavioural learning. Comparing trials matched for preceding simple-spike activity helped rule out that activity alone as the explanation. This connects a separable teaching-related event to later plasticity during behaviour, but extracellular recordings do not directly reveal the numerical update at every parallel fiber synapse, and the finding does not demonstrate multilayer backpropagation.^[ref]

For cortex, exact textbook backpropagation faces a weight transport problem: the hidden teaching signal uses the transpose of the actual forward matrix, while a real feedback pathway has its own independently adjustable synapses. It also needs a way to deliver a target or prediction error, keep learning signals distinct from ordinary activation, and convert them to local plasticity under spiking, delays and sign constraints. The mere presence of anatomical feedback does not establish this computation.

Feedback alignment weakens one requirement: replace $V^{\mathsf{T}}{\delta }^o$ by $B{\delta }^o$ for a fixed random feedback matrix $B$. Simulations showed that changing forward weights can make these approximate teaching directions useful, even without exact reciprocal weights. This is evidence that the weight transport problem need not forbid useful deep error correction; it is not evidence that arbitrary feedback always converges, that the updates remain exact gradient descent, or that this precise mechanism has been recorded in the brain. The work explicitly leaves the physiological implementation and the interaction between feedback activation and learning unresolved.^[ref]

Local supervised error correction has a plausible circuit substrate and direct behavioural-physiological support. Multilayer and recurrent error correction are powerful explanatory possibilities, but their teaching signals, temporal credit assignment and feedback pathways must be established experimentally. Different architectures should be compared by held-out learning behaviour, representational changes and pathway-specific intervention, rather than by labelling all feedback as backpropagation.

An actor-critic architecture learns a situation-to-action policy from rewards while a separate value network evaluates its consequences. It is a concrete reinforcement learning architecture: no supervisor supplies the correct action on each trial. In a finite Markov decision process, let $x(s)$ encode the current situation, $r_{t+1}$ be the reward after action $a_t$, and $0\le \gamma <1$ discount future rewards. The objective is expected return, ${\mathbb{E}}_{\pi }{\sum }_{t\ge 0}{\gamma }^t{r}_{t+1}$. For an episodic undiscounted task, termination can instead make the return finite. State representation matters: if the sensory input does not identify the relevant situation, a memory of previous observations must be added.

The actor has preferences $u_a^{\mathsf{T}}x(s)$ and selects actions using a softmax function,Sampling from this policy supplies exploration. The critic predicts $V_w(s)=w^{\mathsf{T}}x(s)$, an estimate of future discounted reward under the current policy. Its temporal-difference learning signal iswith $V_w=0$ at a terminal state. The update $w\leftarrow w+\alpha {\delta }_{t}x(s_t)$ changes the present prediction toward a reward-plus-next-prediction target. It is a semi-gradient update: it does not differentiate through the bootstrapped target. The temporal-difference principle assigns credit using successive predictions rather than waiting for a complete episode to end.^[ref]

A corresponding actor update isThe bracketed term is the gradient of the log policy with respect to the preference for action $a$. A positive reward prediction error increases the probability of the selected action; a negative one decreases it. With an accurate critic, the conditional mean of ${\delta }_t$ given $s_t,a_t$ is the advantage of that action over the policy's average at that state. This motivates the update, while approximate values, changing policies and limited exploration can compromise learning. It is not a guarantee of the globally best behaviour for every choice of features and learning rates.

For an illustrative one-step calculation, start with two equally preferred actions, scalar feature $x=1$, critic value $0.5$, and a chosen first action followed by reward $1$. Then $\delta =0.5$. With actor learning rate $\beta =0.1$, its two preference changes are $+0.025$ and $-0.025$, so the next probability of the rewarded action is $1/(1+e^{-0.05})\approx 0.5125$. These are chosen model values. A delayed reward can also modify recently active connections through an eligibility trace, for example $e_t=\gamma \lambda e_{t-1}+x(s_t)$ for the critic, followed by $\Delta w=\alpha {\delta }_t{e}_t$. The actor needs its own policy-gradient eligibility. Traces must be reset or otherwise handled at episode boundaries.

The key biological hypothesis is a three-factor plasticity rule: presynaptic activity, postsynaptic or action-selection activity, and a modulatory reward prediction error. Cortical inputs to the striatum can provide situation features, action-related populations can supply the actor, and dopamine can modulate plasticity. This is a functional correspondence, not a one-to-one identification of all actor units with one anatomical compartment. The critic itself must be learned; delivering reward directly to every active connection would not explain prediction-dependent changes.

Recordings in monkeys trained through successive delayed-response tasks provide relevant primary evidence. Phasic responses of putative dopamine neurons to liquid reward were more common during learning than after task performance had become established, while significant stimuli continued to elicit responses. This is compatible with a learning signal sensitive to predictability rather than a permanent readout of reward consumption, but the study also emphasizes attentional and motivational interpretations.^[ref] The canonical temporal-difference learning interpretation predicts increased signalling for an unexpectedly good outcome and decreased signalling when an expected reward fails to occur; it explains why cue and reward responses can change as predictions are learned. This is a quantitative interpretation of physiological evidence, not an assertion that all dopamine activity equals one scalar critic error.^[ref]

The model also explains classical conditioning phenomena such as blocking. If an established cue already predicts the reward, adding another cue leaves little positive prediction error, so the new cue acquires little value. More causally informative experiments in rats stimulated dopamine neurons at reward delivery in blocking and extinction procedures. The stimulation, timed to mimic a positive prediction error, produced lasting increases in cue-elicited reward seeking. This supports a teaching role for phasic dopamine, rather than merely a correlation with successful behaviour. It does not identify every synapse or prove that a complete actor-critic algorithm is the only explanation.^[ref]

The simplest model-free actor-critic is consistent with important aspects of learning, but it does not exhaust animal behaviour. It gradually caches action preferences. It cannot immediately change an unvisited action's value after an outcome is devalued unless the state or reward representation supplies an additional route to that change. In an instrumental conditioning experiment in rats, lever pressing trained on interval schedules became insensitive to sucrose devaluation in sham controls, whereas dorsolateral-striatal lesions preserved sensitivity to devaluation. Consumption controls showed that devaluation itself was effective, and testing in extinction limited new reward learning. This dissociation supports a distinction between habitual and goal-directed control, rather than one universal cached policy.^[ref]

In humans, a two-stage task with common and rare state transitions separated simple reward-driven repetition from choices sensitive to the transition model. Behaviour and striatal prediction-error-related imaging signals showed both model-free reinforcement learning and model-based reinforcement learning influences. The imaging signal is not a direct measurement of individual dopamine spikes, and task performance alone does not uniquely identify a circuit, but it argues against treating every striatal value signal as purely model-free.^[ref]

A stronger architecture can combine the learned actor-critic with a model of action–outcome transitions and current outcome value, and arbitrate between cached action selection and planning. The model's strength is its explicit account of exploration, delayed credit and prediction-dependent plasticity; its limits become visible in devaluation, latent learning and rapid replanning.