Reinforcement learning

Learning to choose actions from their consequences in an environment, with the objective of increasing expected reward or return. Reward does not normally specify a correct action label. State representation, exploration and temporal credit assignment are central computational requirements.

A learning rate controls the size of an iterative learning update. In gradient descent, it scales a direction obtained from the gradient of a loss:A stochastic gradient descent update uses an estimated gradient in this expression. The documented SGD algorithm makes the learning rate an explicit multiplier; momentum and other modifications add separate behavior.^[ref]

A scalar example shows why size matters. For $L(\theta )={\theta }^2/2$, the gradient is $\theta$, so a constant learning rate givesFor nonzero initial value, convergence to zero occurs exactly when $|1-\eta |<1$, or $0<\eta <2$. With ${\theta }_0=4$ and $\eta =1/2$, the values are $4,2,1,1/2,\dots$. With $\eta =5/2$, they begin $4,-6,9,-13.5,\dots$: increasing the step has made the iteration diverge rather than merely learn faster.

This threshold belongs to the stated quadratic. Changing curvature changes the allowed scale: for $L(\theta )=a{\theta }^2/2$ with $a>0$, the same proof gives $0<\eta <2/a$. A numerical learning rate therefore has meaning only together with the objective and update rule.

In reinforcement learning, an incremental value estimate can instead use the update $Q\leftarrow Q+\alpha (r-Q)$. For an illustrative $Q=2$, observed target $r=4$ and $\alpha =0.1$, the new estimate is $2.2$. Here the learning rate weights new evidence against the previous estimate, showing the same step-size idea without requiring a gradient calculation.

A learning rate schedule can vary the rate with iteration. Report the schedule, objective scaling and other update settings when comparing results; the rate alone does not determine stability or final accuracy.

A learning rate schedule specifies how the learning rate changes during an iterative learning procedure. Its rule can depend on the update count, a training epoch count or another declared event. The rule and the clock must both be stated: changing a rate every parameter update is different from changing it after a complete pass through the training data.

A simple step-decay example iswhere $t$ counts completed schedule advances from zero. The values at $t=0,1,2$ are $0.1$, those at $t=3,4,5$ are $0.01$, and the value at $t=6$ is $0.001$. The StepLR implementation documents periodic multiplication by a specified factor; its example advances the scheduler after the training and validation operations.^[ref]

A different schedule admits an exact convergence calculation. For gradient descent on $L(\theta )={\theta }^2/2$, take ${\eta }_t=1/(t+2)$. With no other update terms,Multiplying the successive ratios gives a telescoping product:Starting at ${\theta }_0=4$ gives $4,2,4/3,1,\dots$. This proves convergence for the stated deterministic quadratic, rather than for every objective or noisy learning procedure.

The underlying stochastic gradient descent update also depends on its gradient estimate and any additional terms, not just the scheduled scalar.^[ref] Report the initial index, update order and any external changes to the rate. The StepLR documentation explicitly permits other changes alongside its own, so the isolated step formula above assumes that no competing rule intervenes.^[ref]

A cosine learning rate schedule varies the learning rate along a segment of a cosine curve. For a single decreasing segment, declare $t$ to count completed schedule advances and choose a duration $T>0$. With no competing rate updates, setIt starts at ${\eta }_{\max }$, reaches the midpoint of the two bounds at $T/2$, and ends at ${\eta }_{\min }$. This is the cosine segment used in SGDR; repeating such segments with restarts is an additional policy, not part of the single-segment formula.^[ref]

For an original example, let ${\eta }_{\max }=0.1$, ${\eta }_{\min }=0.01$ and $T=4$. At $t=0,1,2,3,4$ the rates are approximately $0.1,0.08682,0.055,0.02318,0.01$. Differentiating the continuous interpolation givesThus the rate decreases throughout the interior when the upper bound exceeds the lower bound, and its interpolation has zero slope at both ends. A smooth rate curve does not by itself prove convergence of the learning procedure.

The clock must be declared: advancing once per training epoch differs from advancing after every parameter update. The PyTorch CosineAnnealingLR documentation explicitly distinguishes its implementation without restarts from SGDR. Its example advances the scheduler after the training and validation operations. To stop at the lower bound, stop or hold the schedule at the declared endpoint; indefinitely extending the cosine formula is not a permanently decreasing rule.^[ref]

Learning about a distribution of possible future returns rather than only its mean. Multiple prediction channels can vary in optimism and sensitivity to positive and negative errors. Such population heterogeneity can refine a reward prediction error account without implying that every dopamine signal performs the same computation.

Using stored or regenerated experience for further learning, often interleaved with new observations. It can improve data reuse and retention, but its sampling distribution and fidelity affect what objective is learned. A software replay buffer is not a complete biological model of hippocampal reactivation.

Learning or using a model of transitions and outcomes to evaluate possible actions. It can support flexible replanning when the value of an outcome changes, but depends on the learned model and computational resources. Behavioural experiments must distinguish it from alternative strategies with similar observable predictions.

Learning cached values or action preferences without explicitly planning through a model of state transitions and outcomes. A basic actor-critic is an example. The absence of a transition model can limit rapid revaluation of an unvisited action after its outcome changes.

A value prediction is updated using a reward and a subsequent prediction: ${\delta }_t=r_{t+1}+\gamma V(s_{t+1})-V(s_t)$. The linear semi-gradient rule is $\Delta w=\alpha {\delta }_{t}x(s_t)$ for $V_w(s)=w^{\mathsf{T}}x(s)$; the target is held fixed for this update. Terminal-state value is zero in an episodic task.^[ref]

A sampled transition updates only the visited value: $V(S_t)\leftarrow V(S_t)+{\alpha }_t{\delta }_t$, where ${\delta }_t=R_{t+1}+\gamma V(S_{t+1})-V(S_t)$. Under the fixed policy, its conditional mean is $(T^{\pi }V)(s)-V(s)$. Standard tabular convergence requires adequate state visitation and step-size/noise conditions; this mean identity alone does not prove convergence for arbitrary function approximation.

A discrepancy between obtained or newly predicted reward and the reward previously expected. In a temporal-difference model it includes the next-state prediction, rather than being simply the immediate reward. A positive value signals a better-than-predicted outcome; a negative value signals a worse-than-predicted one.

An actor learns an action-selection policy while a critic learns a value prediction used to evaluate outcomes. A temporal-difference error can update both. With an accurate on-policy state-value function, its conditional mean given a state and action is that action’s advantage, supplying a useful policy-learning direction without a labelled correct action.