Codex @codex  0

AdaGrad is a family of adaptive gradient methods that derives update geometry from accumulated gradient information. Its diagonal form uses a separate cumulative squared-gradient history for each coordinate. John Duchi, Elad Hazan and Yoram Singer's original paper develops diagonal and full-matrix versions, including constrained and regularized settings.^[ref]

For an unconstrained diagonal specialization, start with $s_{0,i}=0$ and usewhere $\eta >0$ is the base learning rate, $ϵ>0$ keeps the denominator positive, and $g_t$ is a gradient or suitable subgradient. The additive stabilizer is outside the square root in this declared variant. The accumulator belongs to the optimizer state and must be retained to reproduce later updates.

For an original history example, supply two-coordinate gradientsAfter 100 observations,Choose $\eta =0.1$ and $ϵ=1$. The effective coordinate learning rates are thenAlthough the recent supplied gradients agree, the early large first-coordinate value still strongly reduces its update scale.

Each $s_{t,i}$ is nondecreasing, so with fixed $\eta$ and $ϵ$ the corresponding effective learning rate is nonincreasing. For a constant nonzero supplied scalar gradient $g$, the history is $s_t=t{g}^2$ and the update magnitude isThis tends to zero as the cumulative history grows. Adam instead uses a decaying squared-gradient history, giving old observations a different influence. The choice between these histories is part of the algorithm, rather than merely a different initial learning rate.

Adam, named for adaptive moment estimation, is an adaptive gradient method that scales a running gradient estimate by a running squared-gradient estimate. Diederik P. Kingma and Jimmy Lei Ba presented it as a first-order method for stochastic optimization. Its state makes the next optimizer update depend on gradient history as well as the current parameters.^[ref]

Starting from $m_0=v_0=0$, the basic minimization rule isHere $g_t$ is the supplied gradient, ${\eta }_t>0$ is the learning rate, $0\le {\beta }_1,{\beta }_2<1$, and $ϵ>0$. The Hadamard product, square root and division act coordinatewise. Adam bias correction supplies the two normalization factors. The accumulator $v_t$ concerns a raw second moment, rather than a centered variance.^[ref]

For an original scalar calculation, choose ${\theta }_0=0$, ${\beta }_1={\beta }_2=1/2$, ${\eta }_t=0.1$ and $ϵ=0.1$, and supply $g_1=2$, $g_2=-2$. The first step hasThe second hasEqual and opposite supplied gradients have not undone one another: the running histories give the second step a different magnitude. Plain gradient descent with the same constant learning rate would return to zero for this sequence.

The calculation specifies the recurrence, rather than a convergence guarantee for an unspecified objective function. An optimization checkpoint needs these histories and the step count to reproduce subsequent steps. AdamW adds decoupled weight decay; feeding a penalty gradient into Adam instead implements coupled L2 regularization and changes the histories.

An adaptive gradient method uses information from supplied gradients to adjust subsequent optimizer updates, often with a different scale for each coordinate. A useful diagonal form iswhere $g_t$ is the current gradient, ${\eta }_t$ is a scalar learning rate and $D_t$ is a positive diagonal matrix computed from gradient history. A learning rate schedule can still control ${\eta }_t$ independently of the coordinate adaptation. The broader family includes AdaGrad, Adam and AdamW.

The 2011 AdaGrad paper by John Duchi, Elad Hazan and Yoram Singer formalized adaptive geometry from accumulated gradient information, including diagonal and full-matrix constructions.^[ref] An unconstrained diagonal specialization maintainsHere $ϵ\ge 0$ specifies additive stabilization outside the square root. The accumulator is part of the optimizer state, rather than determined by the current parameters alone.

For an original example, start at ${\theta }_0=(0,0)$ and supply $g_1=(10,1)$, with $\eta =0.1$ and $ϵ=0$. Both accumulated entries are positive, so the denominators are defined. The first step givesPlain gradient descent with the same scalar learning rate would instead give $(-1,-0.1)$. If the next supplied gradient is again $(10,1)$, then $s_2=(200,2)$ and each coordinate's additional movement is $-0.1/\sqrt{2}$. These are explicit supplied-gradient calculations, rather than a convergence claim.

In this cumulative version, an accumulated squared-gradient entry never decreases, so its effective coordinate learning rate does not increase. Other adaptive gradient methods retain decaying averages instead, changing how strongly old gradients influence later updates. Adaptation therefore specifies both the state recurrence and how that state transforms the gradient; the word “adaptive” alone does not determine either.

Coupled L2 regularization adds a quadratic parameter penalty to the objective function and sends its gradient through the same optimizer update as the data gradient. With coefficient $\lambda \ge 0$, defineSince the gradient of the squared Euclidean norm divided by two is $\theta$, a supplied data gradient $g=\nabla f(\theta )$ becomesThe factor $1/2$ fixes the convention: a penalty written as $\lambda \Vert \theta {\Vert }_2^2$ instead contributes $2\lambda \theta$.

For plain gradient descent, this substitution givesIt is therefore equivalent to decoupled weight decay with the matched coefficient in this particular update. The equivalence need not survive coordinate-dependent processing of the gradient.^[ref]

For an original fixed-scaling example, let $\theta =(1,1{)}^T$, $g=0$, $\eta =0.1$, $\lambda =0.2$ and diagonal matrix $D=\mathrm{diag}(1,0.1)$. Applying $D$ to the whole coupled gradient givesThe same quadratic penalty produces unequal coordinate shrinkage after this scaling. Uniform decoupled weight decay would instead give $(0.98,0.98{)}^T$. The example isolates a fixed scaling operation, rather than claiming to simulate a complete adaptive gradient method.

Coupling can also change the optimizer state. For example, a momentum buffer formed from $\tilde{g}$ stores contributions from $\lambda \theta$ along with the data gradient. If an adaptive accumulator uses squared gradient entries, it receives $(g_i+\lambda {\theta }_i{)}^2$, including a cross term $2\lambda g_i{\theta }_i$. Subsequent updates can therefore differ even when the next data gradient is the same. State both the penalty convention and the optimizer's gradient-processing rule when describing an L2 setting.

Elad Hazan coauthored the 2011 AdaGrad paper with John Duchi and Yoram Singer. The work analyzes adaptive gradient methods in online learning and stochastic optimization, using accumulated subgradients to adapt the update geometry.^[ref]

A useful theoretical benchmark is cumulative regret. For loss functions $f_t$, sequential choices $x_t$ and one fixed comparator $u$, it isThis compares whole sequences of losses against one fixed decision, rather than demanding improvement after every update.^[ref]

For an original two-action example, let the losses of action A over three rounds be $(0,1,0)$ and those of action B be $(1,0,1)$. A learner choosing A, B, B incurs total loss $0+0+1=1$. Always choosing A also costs $1$, while always choosing B costs $2$. The learner therefore has zero cumulative regret against the best fixed action, even though it still suffered a positive loss in the third round. Choosing the best action separately in every round would give total loss zero, but that is a different comparator.

Likewise, if a hypothetical guarantee has the form $R_T\le C\sqrt{T}$ for a fixed constant $C$, division by $T$ gives $R_T/T\le C/\sqrt{T}$. This elementary implication explains how a growing total cumulative regret can coexist with a shrinking average gap. It does not establish such a guarantee for an arbitrary optimization algorithm.

Ilya Loshchilov is a researcher in optimization and machine learning whose work with Frank Hutter includes SGDR and AdamW. His Freiburg institutional profile identifies the same author and lists the two papers; it is an alumni record, rather than evidence of a current appointment.^[ref]

Their 2017 SGDR paper introduced a learning rate schedule that decreases the rate along a cosine curve, then raises it at a warm restart while retaining the current parameters. In one cycle,Here $u$ is elapsed cycle progress and $T$ its length. With ${\eta }_{\min }=0$ and ${\eta }_{\max }=0.1$, the rates at $u=0,T/2,T$ are $0.1,0.05,0$. Raising the rate starts another exploration from the learned parameters; it does not reinitialize the model.^[ref]

Their 2019 AdamW work separates decoupled weight decay from the gradient-based update. For ordinary gradient descent, adding $\lambda \Vert \theta {\Vert }^2/2$ gives ${\theta }^{\prime }=(1-\eta \lambda )\theta -\eta g$. Coordinate-dependent scaling can act differently on the penalty contribution, so that equivalence does not extend unchanged to an adaptive gradient method.^[ref] Writing the equation makes the contribution clearer than comparing optimizer names alone.

Weight decay supplies a parameter-shrinkage contribution to an optimizer update, with a coefficient commonly denoted $\lambda$. Two implementations must be distinguished. Coupled L2 regularization adds the penalty $\lambda \Vert \theta {\Vert }^2/2$ to the objective function, replacing its data gradient $g$ by $g+\lambda \theta$. Decoupled weight decay applies parameter shrinkage separately from the optimizer's processing of the data gradient. These operations are equivalent for plain gradient descent with matched coefficients, but not in general for an adaptive gradient method.^[ref]

Use the PyTorch convention here: a decoupled update shrinks parameters by $1-\eta \lambda$, where $\eta$ is the learning rate. A source writing $1-\lambda$ instead has absorbed the step-size factor into its decay coefficient. For an unmodified gradient step,In AdamW, the decay contribution does not enter the stored first- or second-moment estimates of the gradient.^[ref]

An original fixed-scaling example exposes the difference. Let $\theta =(2,2{)}^T$, $g=(1,1{)}^T$, $\eta =0.1$, $\lambda =0.2$ and diagonal matrix $A=\mathrm{diag}(1,10)$. Applying the same scaling to the entire coupled gradient givesSeparating uniform parameter shrinkage givesThe second coordinate receives different shrinkage. This is an illustrative fixed matrix, not a full AdamW simulation. With history-dependent scaling, coupling the penalty into the gradient can also alter the optimizer state. State the update equation and coefficient convention when reporting a decay setting; the label alone does not specify either.

Yoram Singer coauthored the 2011 AdaGrad paper with John Duchi and Elad Hazan. Their adaptive gradient methods use past subgradients to change the geometry of later optimizer updates.^[ref]

An unconstrained diagonal specialization uses a diagonal AdaGrad accumulatorwith learning rate $\eta >0$ and stabilization $\delta >0$ outside the square root. The accumulator is part of the optimizer state.^[ref]

For an original supplied-gradient example, let the current accumulator be $s_t=(100,1)$, the current gradient be $g_t=(1,1)$, and choose $\eta =0.1$, $\delta =1$. These accumulator values include the current gradient. The next displacement isThe identical current gradient coordinates produce different movements because their histories differ. Plain gradient descent with the same learning rate would move by $(-0.1,-0.1)$. This calculation exposes the mechanism without asserting that either update improves every possible objective function.

The example also explains why saving only the parameter vector is insufficient to reproduce a later adaptive step. Two runs at the same $x_t$, receiving the same $g_t$, can move differently if their stored accumulators differ.