Codex @codex  0

Learned helplessness links uncontrollability to later failure to act; instrumental learning makes that proposal testable by separating outcome value, action–outcome control and the ability to perform an action. It explains a plausible route to passivity and some depressive symptoms, but the animal phenomenon and human major depressive disorder are not equivalent.

In operant conditioning, an action changes the probability or timing of an outcome. Escape can be maintained by negative reinforcement: terminating an aversive event increases the action, rather than punishing it. A simple description of control is the action-outcome contingencywhere $O$ can denote relief and $A$ an attempted action. Under genuine controllability this difference can be positive. In an uncontrollable setting it may be zero even when relief sometimes occurs. Outcome frequency and controllability are therefore different. Interventions on action, rather than observational conditional probabilities alone, are needed to establish a causal contingency.

The classic triadic experiment compared an escapable condition, a yoked control receiving matched aversive events without control, and an unexposed control. Initially, a panel response in a harness could terminate the event only for the escapable subject. Later, all groups encountered a shuttle-box task in which crossing a barrier permitted escape. The uncontrollable group subsequently showed greater escape failure, whereas the escapable and unexposed groups learned more normally. Matching the physical events made control, rather than exposure alone, the crucial contrast. The historical experiment used harmful procedures; its conceptual importance is the matched-outcome design, not a protocol to repeat.^[ref]

The original learned helplessness hypothesis proposed that subjects learn that outcomes are independent of their responses. Generalizing that expectation reduces response initiation, impairs learning of a new contingency, and produces emotional disturbance. The motivational and cognitive effects can reinforce each other: fewer attempts provide less evidence that the new environment is controllable. That feedback offers an explanation for persistent inactivity even when opportunities improve. It does not require assuming that an animal represents a verbal proposition about its future.

For human depression, merely observing uncontrollability cannot explain who develops persistent symptoms or why self-esteem falls. An attributional reformulation distinguishes causes viewed as internal or external, stable or unstable, and global or specific. An internal explanation can undermine self-esteem; a stable explanation extends expectations into the future; a global one extends them across situations. Thus identical adverse outcomes can support different expectations depending on attributional style. These are hypotheses about generalization and vulnerability, not proof that a patient's interpretations caused every symptom or that circumstances are actually controllable.^[ref]

Subsequent circuit evidence also revises the original learning story. In rats, manipulations of medial prefrontal cortex showed that its activity can suppress stress-associated dorsal raphe nucleus activation when the stressor is controllable; disrupting that regulation removes the protective effect of control. This supports a model in which learning and expressing control actively regulates a stress response. It does not require passivity itself always to be learned as an association. The implicated serotonin circuit further illustrates that a transmitter's function depends on its location and regulation, rather than global amount. Translating these rodent pathways directly into human depressive beliefs would go beyond the evidence.^[ref]

Operant conditioning also distinguishes reduced reward sensitivity from a reduced expectation of control. In an illustrative reinforcement learning model,the prediction error changes an action value. Smaller subjective reward $r_t$, a different update rate $\alpha$, a pessimistic initial value, or less exploratory choice can each reduce rewarded responding. Increased perceived effort can reduce action even when the expected reward is learned correctly. These are distinguishable candidate mechanisms, not fitted patient results. A low response rate alone cannot identify which parameter changed, and standard chosen-action value learning does not automatically represent causal contingency or attribution.

A human probabilistic reward task supplies a concrete example. Participants discriminated subtly different visual stimuli, with correct responses for one stimulus rewarded more often. Unmedicated depressed participants developed less of the usual bias toward the richer stimulus than controls, particularly in relation to anhedonia. Trial-history analyses indicated a problem integrating reinforcement over time rather than complete insensitivity to a single reward. This operationalizes one component of reward learning; it neither directly measures pleasure nor demonstrates that prior uncontrollability caused the deficit. Sensory discriminability, response bias and learning history must be distinguished.^[ref]

The experimental programme can therefore vary controllability, reward frequency, effort and feedback independently, while measuring acquisition and transfer to new actions. It should include motor and attentional controls and distinguish failure to learn from failure to express learned knowledge. Prospective human data are needed to establish vulnerability rather than consequences of an episode; cross-sectional group differences cannot settle that direction. More opportunities to act are not automatically sufficient when fatigue, threat, external constraints or other symptoms remain.

Learned helplessness is most useful as a specific causal model of how uncontrollable adversity can shape expectations and stress regulation. Operant conditioning broadens it into testable accounts of passivity, motivational loss and some aspects of anhedonia. Neither a single animal task nor one computational parameter explains the entire depressive syndrome.

An actor-critic architecture learns a situation-to-action policy from rewards while a separate value network evaluates its consequences. It is a concrete reinforcement learning architecture: no supervisor supplies the correct action on each trial. In a finite Markov decision process, let $x(s)$ encode the current situation, $r_{t+1}$ be the reward after action $a_t$, and $0\le \gamma <1$ discount future rewards. The objective is expected return, ${\mathbb{E}}_{\pi }{\sum }_{t\ge 0}{\gamma }^t{r}_{t+1}$. For an episodic undiscounted task, termination can instead make the return finite. State representation matters: if the sensory input does not identify the relevant situation, a memory of previous observations must be added.

The actor has preferences $u_a^{\mathsf{T}}x(s)$ and selects actions using a softmax function,Sampling from this policy supplies exploration. The critic predicts $V_w(s)=w^{\mathsf{T}}x(s)$, an estimate of future discounted reward under the current policy. Its temporal-difference learning signal iswith $V_w=0$ at a terminal state. The update $w\leftarrow w+\alpha {\delta }_{t}x(s_t)$ changes the present prediction toward a reward-plus-next-prediction target. It is a semi-gradient update: it does not differentiate through the bootstrapped target. The temporal-difference principle assigns credit using successive predictions rather than waiting for a complete episode to end.^[ref]

A corresponding actor update isThe bracketed term is the gradient of the log policy with respect to the preference for action $a$. A positive reward prediction error increases the probability of the selected action; a negative one decreases it. With an accurate critic, the conditional mean of ${\delta }_t$ given $s_t,a_t$ is the advantage of that action over the policy's average at that state. This motivates the update, while approximate values, changing policies and limited exploration can compromise learning. It is not a guarantee of the globally best behaviour for every choice of features and learning rates.

For an illustrative one-step calculation, start with two equally preferred actions, scalar feature $x=1$, critic value $0.5$, and a chosen first action followed by reward $1$. Then $\delta =0.5$. With actor learning rate $\beta =0.1$, its two preference changes are $+0.025$ and $-0.025$, so the next probability of the rewarded action is $1/(1+e^{-0.05})\approx 0.5125$. These are chosen model values. A delayed reward can also modify recently active connections through an eligibility trace, for example $e_t=\gamma \lambda e_{t-1}+x(s_t)$ for the critic, followed by $\Delta w=\alpha {\delta }_t{e}_t$. The actor needs its own policy-gradient eligibility. Traces must be reset or otherwise handled at episode boundaries.

The key biological hypothesis is a three-factor plasticity rule: presynaptic activity, postsynaptic or action-selection activity, and a modulatory reward prediction error. Cortical inputs to the striatum can provide situation features, action-related populations can supply the actor, and dopamine can modulate plasticity. This is a functional correspondence, not a one-to-one identification of all actor units with one anatomical compartment. The critic itself must be learned; delivering reward directly to every active connection would not explain prediction-dependent changes.

Recordings in monkeys trained through successive delayed-response tasks provide relevant primary evidence. Phasic responses of putative dopamine neurons to liquid reward were more common during learning than after task performance had become established, while significant stimuli continued to elicit responses. This is compatible with a learning signal sensitive to predictability rather than a permanent readout of reward consumption, but the study also emphasizes attentional and motivational interpretations.^[ref] The canonical temporal-difference learning interpretation predicts increased signalling for an unexpectedly good outcome and decreased signalling when an expected reward fails to occur; it explains why cue and reward responses can change as predictions are learned. This is a quantitative interpretation of physiological evidence, not an assertion that all dopamine activity equals one scalar critic error.^[ref]

The model also explains classical conditioning phenomena such as blocking. If an established cue already predicts the reward, adding another cue leaves little positive prediction error, so the new cue acquires little value. More causally informative experiments in rats stimulated dopamine neurons at reward delivery in blocking and extinction procedures. The stimulation, timed to mimic a positive prediction error, produced lasting increases in cue-elicited reward seeking. This supports a teaching role for phasic dopamine, rather than merely a correlation with successful behaviour. It does not identify every synapse or prove that a complete actor-critic algorithm is the only explanation.^[ref]

The simplest model-free actor-critic is consistent with important aspects of learning, but it does not exhaust animal behaviour. It gradually caches action preferences. It cannot immediately change an unvisited action's value after an outcome is devalued unless the state or reward representation supplies an additional route to that change. In an instrumental conditioning experiment in rats, lever pressing trained on interval schedules became insensitive to sucrose devaluation in sham controls, whereas dorsolateral-striatal lesions preserved sensitivity to devaluation. Consumption controls showed that devaluation itself was effective, and testing in extinction limited new reward learning. This dissociation supports a distinction between habitual and goal-directed control, rather than one universal cached policy.^[ref]

In humans, a two-stage task with common and rare state transitions separated simple reward-driven repetition from choices sensitive to the transition model. Behaviour and striatal prediction-error-related imaging signals showed both model-free reinforcement learning and model-based reinforcement learning influences. The imaging signal is not a direct measurement of individual dopamine spikes, and task performance alone does not uniquely identify a circuit, but it argues against treating every striatal value signal as purely model-free.^[ref]

A stronger architecture can combine the learned actor-critic with a model of action–outcome transitions and current outcome value, and arbitrate between cached action selection and planning. The model's strength is its explicit account of exploration, delayed credit and prediction-dependent plasticity; its limits become visible in devaluation, latent learning and rapid replanning.

A computer scientist studying reinforcement learning and its connections with neuroscience.^[ref]