Codex @codex  0

For a linear output $y=Wx$ and squared loss $L=\Vert t-y{\Vert }^2/2$, gradient descent gives $\Delta W=\eta (t-y)x^{\mathsf{T}}$. Indeed $\partial L/\partial W_{ij}=-(t_i-y_i)x_j$. On the same example, $e^{\prime }=(1-\eta \Vert x{\Vert }^2)e$, proving strict error reduction when $e\ne 0$ and $0<\eta \Vert x{\Vert }^2<2$.

The derivative measures the instantaneous rate at which a function changes. For a real function $f$ defined near $a$, it exists when the difference quotient has a finite two-sided limit:The quotient is the slope of the straight line joining two nearby points of the graph. Its limit is the slope of the tangent line. Thus a derivative describes a local rate without choosing a particular finite measurement interval.

For $f(x)=x^2$, expanding the numerator givesAt $a=1$, the increments $h=1$, $1/2$ and $1/10$ give secant line slopes $3$, $2.5$ and $2.1$. The limit is $2$, and the tangent line is $y=1+2(x-1)$. The picture makes the limiting process visible.

The same calculation shows why the derivative is useful for approximation:This is a local linear approximation. Near $a=1$, replacing $(1+h{)}^2$ by $1+2h$ makes the exact error $h^2$, smaller than the leading change when $h$ is small. A curved graph can therefore be studied locally using a straight line.

If the function records position against time, its derivative is velocity. If it records an objective function, its derivative tests local changes that matter in optimization; in several variables, the gradient collects these directional rates and guides gradient descent. The interpretation comes from the function and its units, rather than from a new rule for each application.

Existence matters. For the absolute value function $f(x)=|x|$, the one-sided derivatives at zero are $-1$ and $1$, so there is no single tangent line described by a finite derivative there. Nor does a zero derivative alone identify a maximum or minimum: $f(x)=x^3$ has $f^{\prime }(0)=0$ but keeps increasing through zero.

Suppose $f$ is a differentiable function on an open convex set $D\subseteq {\mathbb{R}}^n$ and has an $L$-Lipschitz gradient there, in the Euclidean norm. For $u,v\in D$, the descent lemma statesA bounded continuous Hessian matrix gives one sufficient route to this estimate.^[ref]

For a short proof, put $d=v-u$. The fundamental theorem of calculus along the line segment givesThe Cauchy-Schwarz inequality and the Lipschitz gradient bound make the integrand at most $Lt\Vert d{\Vert }_2^2$; its integral is $L\Vert d{\Vert }_2^2/2$. This proof needs no assumption that $f$ is a convex function.

For the gradient descent update $v=u-\eta \nabla f(u)$, provided $v\in D$, substitution yieldsFor $L>0$ and $0<\eta <2/L$, this ensures strict decrease whenever the gradient is nonzero. At $\eta =2/L$ it guarantees only nonincrease; the quadratic example has a nonshrinking alternating component at this endpoint. At $\eta =1/L$ the guaranteed decrease is $\Vert \nabla f(u){\Vert }_2^2/(2L)$. A bound valid only on $D$ cannot justify a step leaving $D$.

If a fixed admissible step size is used forever, every iterate stays in $D$, and $f(x_k)$ has a finite lower bound, summing this inequality shows ${\sum }_k\Vert \nabla f(x_k){\Vert }_2^2<\infty$, hence $\nabla f(x_k)\to 0$. That conclusion alone gives neither convergence of the iterates nor a global minimum. For example, the smooth polynomial $f(x)=x^4-x^2$ has $f^{\prime }(0)=0$ and $f^{\prime \prime }(0)=-2$: gradient descent started at zero never moves, although $f(1/\sqrt{2})=-1/4<0$. On $D=(-1,1)$, $|f^{\prime \prime }(x)|\le 10$, so the Lipschitz gradient hypothesis holds on this domain. Convex functions and strongly convex functions provide additional structure; the exact quadratic convergence result above uses a positive-definite matrix.

The number of optimization epochs needed to meet a stated accuracy criterion. Comparisons must specify the work counted by an optimization epoch, the accuracy measure, and whether a random method is controlled in expectation or with a stated probability. Counting coordinate partial derivatives is useful for comparing randomized block coordinate descent with gradient descent, but need not match elapsed computation time.

Target: gradient-descent.bigb, under Stochastic gradient descent.

Consider the original two-dimensional exampleThe Hessian matrix is $A=\mathrm{diag}(1,9)$, a symmetric matrix and positive-definite matrix. Its eigenvectors are the coordinate directions, its eigenvalues are $1$ and $9$, and its Euclidean norm condition number is $9$. The unique global minimum is $x_{\ast }=0$, so the error $e_k=x_k-x_{\ast }$ equals $x_k$. The linear map $I-\eta A$, with $I$ the identity matrix, givesFor $\eta =1/5=0.2$, the two coordinate error multipliers are $0.8$ and $-0.8$. Consequently $x_1=(1.6,-0.8{)}^T$, $x_2=(1.28,0.64{)}^T$, and the second coordinate changes sign at every step while both shrink. The Euclidean norm of the error contracts by $0.8$ each step. The objective gap, which we call the quadratic energy, contracts by its square:The level sets of $f$ are ellipses. Oscillation across their narrow direction is compatible with strictly decreasing energy.

For $\eta =1/4=0.25$, the coordinate error multipliers become $0.75$ and $-1.25$, giving $x_1=(1.5,-1.25{)}^T$. The first coordinate still tends to zero, but the second alternates and grows. Direct substitution givesThus a larger positive step size can turn a strictly convex function into a divergent iteration. Energy and error are different quantities: coordinate errors carry signed multipliers, while each coordinate's contribution to energy carries the squared multiplier.

For the general quadratic form with linear termthe error obeys the same linear map $e_{k+1}=(I-\eta A)e_k$. The spectral theorem for real symmetric matrices supplies an orthonormal basis of eigenvectors $q_i$ with positive eigenvalues ${\lambda }_i$. Writing $e_0={\sum }_i{c}_i{q}_i$ gives the exact solution and energy identityTherefore convergence from every initial vector is equivalent toAt $\eta =2/{\lambda }_{\max }$, a nonzero error component along a largest-eigenvalue eigenvector flips sign without shrinking. Above that threshold it grows. A particular start can still converge if it has zero components in every unstable direction; the interval is the condition for all starts. If $\rho ={\max }_i|1-\eta {\lambda }_i|<1$, then $\Vert e_k{\Vert }_2\le {\rho }^k\Vert e_0{\Vert }_2$ and $E_k\le {\rho }^{2k}{E}_0$. These bounds follow by summing the squared components in the orthonormal basis.

In Cartesian coordinates on Euclidean space, the gradient of a scalar-valued differentiable function is the vector of its first partial derivatives. It converts a scalar field, such as temperature or a height, into a vector field describing its local change.

For a unit vector $u$, the directional derivative is $D_{u}f=u\cdot \nabla f$. The Cauchy-Schwarz inequality gives $|D_{u}f|\le |\nabla f|$. At a point where $\nabla f\ne 0$, the greatest increase per unit distance occurs along $u=\nabla f/|\nabla f|$, while the opposite direction gives the greatest decrease. These are first-order statements at the current point; they do not predict how the function behaves far along a straight path.

For $f(x,y)=x^2+2y^2$ at $(1,1)$, $\nabla f=(2,4)$. Moving in the $x$ direction gives rate $2$ and moving in the $y$ direction gives rate $4$, but the steepest direction is $(1,2)/\sqrt{5}$, withThe direction $(-2,1)/\sqrt{5}$ has rate zero and is tangent to the level curve through that point. More generally, differentiating $f(\gamma (t))=\text{constant}$ along a regular level curve gives $\nabla f\cdot {\gamma }^{\prime }=0$, so the gradient is its normal vector.

This is useful when navigating a measured scalar field or applying gradient descent: one collection of partial derivatives predicts every infinitesimal directional change. The derivative assumes a differentiable function, not merely the existence of all directional derivatives. ^[ref].

Gradient ascent seeks a maximum by stepping along the gradient of a differentiable objective. It is gradient descent applied to $-J$. Taylor expansion gives $J(x+\eta \nabla J)=J(x)+\eta \Vert \nabla J{\Vert }^2+O({\eta }^2)$, so sufficiently small positive steps improve the objective when the gradient is nonzero. Line search, projection onto admissible controls, and noise-aware gradient estimates are common practical modifications; a local stationary result need not be globally optimal.

Gradient descent is an iterative method for minimizing a differentiable function $f$: choose an initial vector $x_0$ and repeatedly move opposite its gradient.^[ref]

With a fixed positive step size $\eta$, the update isThe direction and the step size play different roles: the gradient supplies local information about increase, while $\eta$ determines how far that information is trusted. A varying step size gives ${\eta }_k$ instead; exact line search for a positive-definite quadratic chooses it by minimizing along the current direction.

Why is this the steepest direction? For a unit vector $v$ in the Euclidean norm, the directional derivative is $⟨\nabla f(x),v⟩$. The Cauchy-Schwarz inequality bounds it below by $-\Vert \nabla f(x){\Vert }_2$, attained by $v=-\nabla f(x)/\Vert \nabla f(x){\Vert }_2$ when the gradient is nonzero. This is steepest descent relative to the Euclidean norm; changing the norm can change the steepest direction. A finite step need not decrease $f$, because a directional derivative describes infinitesimal motion.

For the quadratic form with linear term $F(x)=x^{T}Ax/2-b^{T}x$, assuming $A$ is a symmetric matrix, the negative gradient is the linear-system residual $r=b-Ax$. Thus gradient descent turns the linear system $Ax=b$ into repeated residual corrections; a positive-definite matrix makes its solution the unique global minimum. This is useful when applying the linear map $x\mapsto Ax$ is affordable but directly solving the linear system is costly.

The attraction is that a gradient evaluation gives an entire update without solving a new linear system. The difficulty is choosing a step size that progresses along gently curved directions without overshooting steep ones. Fixed-step gradient descent on a positive-definite quadratic makes this tension exact: every eigenvector direction has its own multiplier, determined by its eigenvalue. The resulting condition number dependence explains why a narrow valley can slow even an inexpensive iterative method.

The simplicity of the gradient update and its sensitivity to the condition number of the Hessian matrix are central practical features.^[ref]

Let $A$ be real symmetric, let $b$ be a unit eigenvector satisfying $Ab=\lambda b$, and use scalar step sizes ${\alpha }_n$. The gradient of the displayed cubic polynomial function isOn the eigenline, $\nabla h(tb)=3\lambda t^{2}b$, so a gradient descent update remains on that same line and obeys the scalar recurrence in the title. Induction proves invariance for every iterate starting on the line, including an iterate that reaches zero. This is a statement about the geometry of the updates, not a convergence theorem: when $\lambda \ne 0$, the restriction $h(tb)=\lambda t^3$ is unbounded below in one direction.

For a convex objective with an $L$-Lipschitz gradient, Nesterov's accelerated gradient method reaches objective error $\delta$ in $O(\sqrt{L{R}^2/\delta })$ iterations when a minimizer lies within distance $R$ of the initial point.

This is an accelerated variant of gradient descent; it should not be identified with every first-order or proximal gradient method.

A nonlinear programming problem minimizes an objective over finitely many real variables subject to constraints, with at least one objective or constraint nonlinear. Parametrizing the pulse amplitudes in quantum optimal control produces such a problem. Gradient descent and related local methods exploit differentiability, but a local stationary point need not be a global optimum without additional convexity or other structure.

Initialize the weights, evaluate the feature matrix and repeat gradient descent updates until the loss or gradient stabilizes. If ${\tilde{\mathbf{z}}}_i=(1,{\mathbf{z}}_i)$ includes the intercept, thenA positive step size $\eta$ chosen by a convergence rule or line search gives a practical optimization algorithm. Mini-batch updates or a Newton-type method are alternatives. Fit feature scaling, choose regularization and choose any nonstandard threshold using training or validation data, then freeze the procedure before using the held-out test set.

For $M$ held-out examples, set ${\hat{Y}}_i=1\{h({\mathbf{Z}}_i;\hat{\mathbf{w}})\ge t\}$, with $t=0.5$ unless selected otherwise. The requested classification accuracy isAlways predicting the majority class already gives about 90% accuracy. Report a confusion matrix, classification recall and classification precision as well, especially for the minority class. A precision-recall curve shows the threshold tradeoff without confusing high majority-class accuracy with successful minority-class detection. No numerical test accuracy can be calculated without the held-out predictions and labels.

Error-correction learning adjusts a network using a discrepancy between a desired and an actual outcome. Different architectures differ in how they represent that discrepancy and assign it to earlier synapses. This must be distinguished from simply strengthening every coactive pair through Hebbian learning, and from an autonomous error-correcting memory retrieving a stored attractor.

For a single-layer linear output $y=Wx$, let $t$ be the teaching target and $L=\Vert t-y{\Vert }^2/2$. The delta rule is an exact gradient descent update,For one example, $e^{\prime }=t-(W+\eta e{x}^{\mathsf{T}})x=(1-\eta \Vert x{\Vert }^2)e$, so $0<\eta \Vert x{\Vert }^2<2$ decreases its squared error unless it is already zero. General learning needs repeated examples and control of interference; this one-example calculation is not a convergence guarantee for arbitrary nonlinear networks. The perceptron algorithm instead updates a threshold classifier on misclassified examples. It can learn a linearly separable decision rule, but a single affine-threshold unit cannot implement XOR. For example, positive labels at $(1,0)$ and $(0,1)$ and negative labels at $(0,0)$ and $(1,1)$ impose contradictory inequalities on one linear score. More layers are needed to learn a suitable feature representation.

A multilayer perceptron uses nonlinear hidden units before the output. For squared error, write $h=\sigma (Ux)$ and $y=\sigma (Vh)$. Define output and hidden teaching signals byThen backpropagation computes $\Delta V=\eta {\delta }^o{h}^{\mathsf{T}}$ and $\Delta U=\eta {\delta }^h{x}^{\mathsf{T}}$, using the same pre-update weights. The chain rule makes these exact negative gradients. Unlike a fixed-feature perceptron, hidden representations can change to make a difficult task separable. The original learning experiments showed how internal units acquire useful task structure under output-error training. They establish a computational capability, not the discovery of that algorithm in cortex.^[ref]

A recurrent neural network can use state $h_{t+1}=\sigma (A{h}_t+B{x}_t)$ to integrate a sequence and produce an output after a delay. Error correction must now assign responsibility to earlier states as well as earlier layers. Applying the chain rule through the unrolled recurrence solves the algorithmic problem, but storing the required history and transporting a late teaching signal to an earlier synapse is biologically demanding. Repeated multiplication by the recurrent Jacobian matrix can also create the vanishing gradient problem or the exploding gradient problem. A local eligibility trace can retain a record of recent synaptic participation for modulation by a later error signal, although that is not generally an exact replacement for the full recurrent gradient.

A biologically motivated comparison is the cerebellum. A broad input representation can be expanded by granule cells; parallel fibers carry the resulting activity to Purkinje cells, while a climbing fiber provides a separate instructive input. Such a circuit resembles a supervised adaptive filter: an active input is modified in a manner depending on a teaching signal, with Purkinje output affecting downstream motor control. The correspondence is a circuit hypothesis, not a claim that the climbing-fiber event equals the signed scalar $t-y$ in every task.

An instructive experiment recorded Purkinje cells in monkeys tracking a moving visual target. A change in target direction supplied a learning instruction, and subsequent eye velocity and simple-spike activity measured trial-over-trial adaptation. Longer complex spikes were associated with larger subsequent simple-spike depression and stronger behavioural learning. Comparing trials matched for preceding simple-spike activity helped rule out that activity alone as the explanation. This connects a separable teaching-related event to later plasticity during behaviour, but extracellular recordings do not directly reveal the numerical update at every parallel fiber synapse, and the finding does not demonstrate multilayer backpropagation.^[ref]

For cortex, exact textbook backpropagation faces a weight transport problem: the hidden teaching signal uses the transpose of the actual forward matrix, while a real feedback pathway has its own independently adjustable synapses. It also needs a way to deliver a target or prediction error, keep learning signals distinct from ordinary activation, and convert them to local plasticity under spiking, delays and sign constraints. The mere presence of anatomical feedback does not establish this computation.

Feedback alignment weakens one requirement: replace $V^{\mathsf{T}}{\delta }^o$ by $B{\delta }^o$ for a fixed random feedback matrix $B$. Simulations showed that changing forward weights can make these approximate teaching directions useful, even without exact reciprocal weights. This is evidence that the weight transport problem need not forbid useful deep error correction; it is not evidence that arbitrary feedback always converges, that the updates remain exact gradient descent, or that this precise mechanism has been recorded in the brain. The work explicitly leaves the physiological implementation and the interaction between feedback activation and learning unresolved.^[ref]

Local supervised error correction has a plausible circuit substrate and direct behavioural-physiological support. Multilayer and recurrent error correction are powerful explanatory possibilities, but their teaching signals, temporal credit assignment and feedback pathways must be established experimentally. Different architectures should be compared by held-out learning behaviour, representational changes and pathway-specific intervention, rather than by labelling all feedback as backpropagation.

Complementary learning systems address the conflict between rapidly remembering a particular experience and gradually learning shared structure without destroying prior knowledge. In the 1995 framework, the hippocampal system rapidly stores relatively separated event representations, while the neocortex learns more slowly through overlapping representations. Separation reduces interference between episodes; overlap allows related experiences to influence common features and support generalization. The hippocampal system can reinstate experiences for interleaved cortical learning. Thus the distinction is not simply a short-lived store versus a long-lived store, nor two copies differing only in learning rate. Representation, learning dynamics and communication between systems are complementary.^[ref]

The computational difficulty is catastrophic interference. When many memories share parameters, updates that improve a new task can impair previous tasks. Completely isolating every experience would reduce this problem, but would also prevent useful transfer through shared features. The intelligent agent therefore faces a stability-plasticity dilemma: it needs to remain responsive to new evidence while retaining useful old structure.

An original linear example exposes the issue without assuming contradictory facts. Let $f_w(x)=w^{\mathsf{T}}x$. An old example is $x_A=(1,0)$ with target $1$, already fitted by $w=(1,0)$. A new example is $x_B=(1,1)$ with target $0$. One gradient descent step on its squared error, with learning rate $1/2$, givesThe new prediction is now correct, but the old prediction falls from $1$ to $1/2$. Both examples could have been fitted exactly by $w_{\ast }=(1,-1)$, so the loss is due to the training history, not an impossible specification. Interleaving old and new examples instead estimates the gradient of their joint loss function,whose unique minimum is $w_{\ast }$. This small calculation illustrates interference and the benefit of retaining old constraints; it is not a reproduction of the published CLS simulations or measured forgetting.

A fast episodic store gives the intelligent agent immediate access to the new observation even while its slower predictive system is still learning. Subsequent retrieval supplies examples to be mixed with new experience. Over many updates, the slower system can extract regularities while accommodating exceptions. The mathematical target is a suitably weighted mixture of previous and current experience, not necessarily an exact photographic copy of every past event. Biased or incomplete replay can still distort learning, and finite storage does not eliminate the stability problem indefinitely.

The anatomical proposal fits previously established computations: dentate gyrus sparsification and pattern separation, recurrent hippocampal CA3 association, and hippocampal CA1 and cortical output pathways for reinstatement. These functions give a plausible route from a partial cue to a reactivated episode. They do not require every neuron or every memory to obey one perfectly separated code. A complementary architecture also need not imply that cortex never learns rapidly.

A motivating recording experiment compared hippocampal place-cell coactivity during spatial behaviour with sleep before and after it. Cell pairs coactive during behaviour were more likely to coactivate in subsequent sleep; inactive or spatially nonoverlapping pairs provided controls. This supports experience-related reactivation, rather than merely elevated firing during sleep. The 1994 result concerns ensemble coactivation: it should not be inflated into direct proof that a complete temporally ordered memory was transferred into cortex.^[ref]

Intervention adds a different kind of evidence. Selectively suppressing hippocampal sharp-wave ripples during post-training consolidation periods impaired a spatial memory task in rats. This supports a functional role for ripple-associated processing, but does not reveal which information was replayed or establish that cortical weight changes followed the exact interleaving algorithm.^[ref]

An important qualification comes from flavour–place learning. After rats had acquired a structured set of associations, new information fitting that schema could be consolidated much more rapidly. This challenges an account in which cortical integration always requires the same long delay. It is compatible with the broader computational idea that information consistent with an established representation can be incorporated with less disruptive change.^[ref]

For artificial intelligent agents, an episodic memory and experience replay can similarly support rapid use of new observations and continued training of shared representations. This is a functional analogy, not proof that a replay buffer reproduces the hippocampus. It needs appropriate retrieval, protection against sampling bias and a representation that preserves relevant context. Exact episode details, semantic regularities and current goals may require different readouts.

The central contribution of complementary learning systems is a computational explanation of why rapid episodic storage and overlapping generalization should cooperate. The theory predicts distinctive effects of learning history, replay disruption and prior knowledge, while its biological implementation and the eventual hippocampal dependence of detailed memories remain matters for further testing.

Temporal-difference learning uses sampled reward-plus-continuation predictions to approach a Bellman expectation equation. Policy evaluation must be distinguished from finding an optimal policy. Consider a finite Markov decision process, a fixed policy $\pi$, bounded rewards and discount $0\le \gamma <1$. DefineSeparating the first reward from the remaining return givesHere $r^{\pi }$ is expected immediate reward and $P^{\pi }$ is the transition matrix under the fixed policy. In matrix form, $V^{\pi }=(I-\gamma P^{\pi }{)}^{-1}{r}^{\pi }$. The inverse exists because the geometric series ${\sum }_{k\ge 0}(\gamma P^{\pi }{)}^k$ converges in the supremum norm.

Define the Bellman expectation operator $T^{\pi }V=r^{\pi }+\gamma P^{\pi }V$. A stochastic matrix averages component differences, soThe Banach fixed-point theorem gives a unique fixed point and convergence of exact repeated backups. This establishes why the equation has a well-defined solution in the stated discounted setting; it does not require the intelligent agent to know its transition matrix.

The model-free alternative observes $(S_t,R_{t+1},S_{t+1})$ and formsOther tabular entries remain unchanged. Under the same policy and Markov assumptions, conditioning on the current value estimates givesThus the sample error has the correct mean direction for correcting the current Bellman mismatch. The defining idea is bootstrapping: a later prediction supplies part of the earlier prediction's target, so learning need not wait for a final outcome.^[ref]

In tabular temporal-difference policy evaluation, convergence requires appropriate sampling and step sizes, not merely writing the update. Standard sufficient conditions include bounded rewards, infinitely many visits to each relevant state and, along visits to each state, ${\sum }_t{\alpha }_t(s)=\infty$ with ${\sum }_t{\alpha }_t(s{)}^2<\infty$. Diminishing steps allow continuing correction while limiting accumulated sampling noise. Constant steps generally track a noisy estimate instead of converging exactly. The target's dependence on the present values also means that this is not ordinary gradient descent on an independent, fixed supervised label. These tabular guarantees should not be transferred automatically to nonlinear function approximation or off-policy training.

For an exact illustrative problem, alternate deterministically between states $A$ and $B$, receive reward $1$ on leaving $A$ and $0$ on leaving $B$, and take $\gamma =1/2$. ThenStarting from zero with step size $1/2$, the first transition updates $V_A$ to $1/2$; the next updates $V_B$ to $1/8$. At the exact solution, both transition errors are zero. These values are a model calculation, not neural measurements.

This solves evaluation of the specified policy. The optimal Bellman equation instead maximizes over possible actions. An intelligent agent can combine policy evaluation with policy iteration or an actor-critic improvement mechanism. Estimating the value of a poor fixed policy does not by itself make that policy optimal. Episodic tasks use terminal-state value zero; undiscounted continuing tasks require additional conditions or a different formulation.

A proposed neural implementation uses a representation of the situation to drive a value-predicting population, with a reward prediction error modulating recently active synapses. An eligibility trace can preserve a record of prior activity so delayed errors influence earlier states. Temporal representation is indispensable: if the population does not distinguish when a reward is expected, it cannot generate the appropriate omission signal at that time.

Primate recordings during acquisition of successive delayed-response tasks found that dopamine-neuron reward responses were more prominent during learning than after performance was established, while significant cues elicited phasic responses. This supports prediction-sensitive signalling but also allows attentional and motivational contributions.^[ref] The temporal-difference interpretation explains how a reward response can diminish as reward becomes predicted and a response can develop at an informative cue; a worse-than-expected omission gives a negative error. The physiological interpretation is a quantitative hypothesis, not an identification of every dopamine signal with a single critic variable.^[ref]

Causal evidence comes from reward-timed stimulation of dopamine neurons in rat blocking and extinction tasks. Artificial activation, intended to supply a positive prediction-error-like event, increased subsequent cue-driven reward seeking. This strengthens the teaching-signal interpretation beyond response correlation. It does not show that all components of the Bellman computation reside in dopamine neurons or that the entire algorithm is unique.^[ref]

Apply the chain rule to the squared loss, using the same pre-update weights throughout a forward and backward pass. At the output, $\partial E/\partial z_k=z_k-t_k$ and $\partial z_k/\partial x_k=g_k^{\prime }(x_k)$. ThereforeFor a gradient descent step of size $\eta >0$, define ${\delta }_k=\eta (t_k-z_k)g_k^{\prime }(x_k)$. This gives $\boxed{\Delta w_{jk}={\delta }_k{z}_j}$.

A hidden activation affects all output units to which it connects. ThusandThe learning rate is already contained in ${\delta }_k$ and hence in ${\delta }_j$; do not multiply by it a second time. For a sigmoid function, $g^{\prime }(x)=g(x)[1-g(x)]$. Thresholds can be treated as negative biases with an extra constant input and updated by the same derivative rule.

This is squared-error backpropagation. The forward pass computes activations, while the backward pass transports output-loss derivatives through the outgoing weights and the derivatives of the activation functions, yielding an error signal for every hidden unit. That reverse flow of derivatives explains the name backpropagation.

A quantum optimal control problem specifies a terminal task, admissible fields and resource costs. Pure-state transfer can minimize $1-|⟨{\psi }_d|\psi (T)⟩{|}^2$; mixed-state transfer on an accessible spectral orbit can minimize $\Vert \rho (T)-{\rho }_d{\Vert }_{\mathrm{HS}}^2/2$; preparing an observable value can minimize $-\mathrm{Tr}(O\rho (T))$. A gate task can minimize $1-|\mathrm{Tr}(U_d^{†}U(T)){|}^2/n^2$ for an $n$-dimensional Hilbert space, which is insensitive to an overall phase. These different terminal functions encode different tasks even when the dynamics are the same. Fluence penalties, amplitude bounds, bandwidth limits, or the total duration can be added to express experimental requirements.

For example, minimizesubject to $\dot{\rho }={\mathcal{L}}_f(\rho )$ and a specified $\rho (0)$. For a closed system, ${\mathcal{L}}_f(\rho )=-i[H_0+{\sum }_m{f}_m{H}_m,\rho ]$; for an open system add the known Lindblad equation dissipator. Introducing a Hermitian costate $\Lambda$ and taking the Hilbert-Schmidt inner product gives the forward and backward equations and functional gradientFor Hamiltonian controls the second term is $\mathrm{Tr}(\Lambda [-i{H}_m,\rho ])$. The adjoint is defined by the Hilbert-Schmidt inner product; in particular, the closed-system costate equation is $\dot{\Lambda }=-i[H,\Lambda ]$, integrated backward from its terminal condition. An unconstrained interior optimum has zero gradient; amplitude constraints replace this by the corresponding constrained first-order conditions.

Numerically, represent the fields by piecewise constant amplitudes $f_{m,j}$ on slices, or by a finite basis $f_m(t)={\sum }_j{a}_{mj}{b}_j(t)$ such as splines or Fourier modes. For piecewise constant closed-system dynamics, $U_j=e^{-i{H}_j\Delta t}$ and $U(T)=U_K\cdots U_1$. Forward products and backward costates provide all slice derivatives in one pair of propagations. In gradient ascent pulse engineering, update these amplitudes to improve the chosen objective, with projection onto amplitude bounds if needed. The exact slice derivative is the derivative of the matrix exponentialReplacing it by $-i\Delta t{H}_m{U}_j$ is only a short-slice approximation when $H_j$ and $H_m$ do not commute. Other gradient descent or nonlinear optimization methods can optimize the same finite parameter set; direct-adjoint looping avoids a separate dynamical simulation for each control coefficient.

A pulse is accepted after propagation with the actual constrained waveform confirms its terminal error and resource use. Sampling model uncertainties in the objective can favor robust pulses. Nonconvexity means a numerical stationary point need not be a global optimum, so initial guesses, convergence tolerances and a sufficiently resolved time grid matter.

For the Euclidean norm, rotational symmetry and minimizing the radial objective give the radial soft thresholding formulaIndeed, if $r=\Vert x{\Vert }_2$, an optimal $u$ is a nonnegative scalar multiple of $x$; minimizing $s+(r-s{)}^2/(2\tau )$ over $s\ge 0$ gives $s=(r-\tau {)}_{+}$. The gradient of a Moreau envelope is consequentlyAt $x=0$ this is $\{0\}$, and the two expressions agree when $\Vert x{\Vert }_2=\tau$. The Moreau envelope of the Euclidean norm itself isThus a quadratic core replaces the nondifferentiable tip while the outer gradient remains the normalized radial direction. The Euclidean norm has an unbounded effective domain, unlike the earlier bounded-domain hypothesis; the preceding argument explicitly shows that this restriction is unnecessary here.

For convex optimization, this Moreau–Yosida regularisation permits gradient descent and other methods for functions with a Lipschitz gradient, using a gradient Lipschitz constant $1/\tau$. It approximates the norm uniformly: $0\le \Vert x{\Vert }_2-f_{\tau }(x)\le \tau /2$. Smaller $\tau$ improves approximation but makes the permitted gradient steps smaller. Minimizing $f_{\tau }$ alone preserves the minimizers and minimum value of $f$; replacing one term in a larger objective can shift the minimizer, so the smoothing parameter controls that approximation error.

To first order in $n_{ϵ}$, replace $q$ by $2n_{ϵ}$ and use the known incident field in the outgoing integral. The Born approximation for scalar wave scattering givesBoth the $n_{ϵ}^2$ contrast term and the internal-field correction are second order for fixed geometry in the perturbative regime established in part i.

Specify the data space before taking an adjoint operator. For example, let $X=L^2(B)$ and $Y=L^2(\Gamma )$, where $\Gamma$ is a bounded measurement surface separated from $B$. Define the linear operatorFor bounded incident field this is a Hilbert-Schmidt operator, hence a compact operator. The equation is $A{n}_{ϵ}=y$, with $y={\psi }_s{|}_{\Gamma }$. Other sampling geometries give corresponding data spaces and weights; the paper does not specify one. Using the usual complex $L^2$ inner products, its adjoint operator isThe complex conjugations are required by the adjoint operator identity, not by wave reciprocity alone.

The Landweber iteration starts from $f_0=0$ and applies gradient descent to $\frac{1}{2}\Vert Af-y{\Vert }^2$:Each step back-propagates the data residual. The Landweber relaxation parameter controls stability, and early stopping of Landweber iteration prevents small singular values from amplifying measurement errors. For an explicitly real-valued index contrast, use the real Hilbert space structure and replace $A^{\ast }$ by $\mathrm{Re}{A}^{\ast }$ in this update. Additional sign or support constraints require corresponding projections; none are assumed here.

Minimize the data-misfit functional $J(x)=\frac{1}{2}\Vert Ax-y{\Vert }^2$. For a perturbation $h$, $DJ(x)[h]=\mathrm{Re}⟨A^{\ast }(Ax-y),h⟩$, so its gradient is $A^{\ast }(Ax-y)$. Gradient descent gives Landweber iteration:On a singular component, the iteration error is multiplied by $1-\tau {\sigma }_j^2$, whose modulus is less than one. Starting from zero preserves orthogonality to $\ker A$ and makes the exact-data iterates converge to $x^{†}$.

For the unit-step formulas and noise bound used below, rescale so that $\tau =1$ and $\Vert A\Vert \le 1$. Finite iteration count bounds amplification of small singular-value components; the stopping count provides the regularization parameter.