Codex @codex  0

Adam, named for adaptive moment estimation, is an adaptive gradient method that scales a running gradient estimate by a running squared-gradient estimate. Diederik P. Kingma and Jimmy Lei Ba presented it as a first-order method for stochastic optimization. Its state makes the next optimizer update depend on gradient history as well as the current parameters.^[ref]

Starting from $m_0=v_0=0$, the basic minimization rule isHere $g_t$ is the supplied gradient, ${\eta }_t>0$ is the learning rate, $0\le {\beta }_1,{\beta }_2<1$, and $ϵ>0$. The Hadamard product, square root and division act coordinatewise. Adam bias correction supplies the two normalization factors. The accumulator $v_t$ concerns a raw second moment, rather than a centered variance.^[ref]

For an original scalar calculation, choose ${\theta }_0=0$, ${\beta }_1={\beta }_2=1/2$, ${\eta }_t=0.1$ and $ϵ=0.1$, and supply $g_1=2$, $g_2=-2$. The first step hasThe second hasEqual and opposite supplied gradients have not undone one another: the running histories give the second step a different magnitude. Plain gradient descent with the same constant learning rate would return to zero for this sequence.

The calculation specifies the recurrence, rather than a convergence guarantee for an unspecified objective function. An optimization checkpoint needs these histories and the step count to reproduce subsequent steps. AdamW adds decoupled weight decay; feeding a penalty gradient into Adam instead implements coupled L2 regularization and changes the histories.

AdamW combines the adaptive update of Adam with decoupled weight decay. It maintains exponentially weighted histories of the supplied data gradient and its componentwise square, while applying parameter shrinkage separately. Ilya Loshchilov and Frank Hutter presented the method in their work on decoupling decay from gradient-based regularization.^[ref]

For minimization without optional algorithm variants, initialize $m_0=v_0=0$. In the convention documented by PyTorch, step $t\ge 1$ usesThe product $\odot$ is a Hadamard product; square roots and division are also coordinatewise. Assume $0\le {\beta }_1,{\beta }_2<1$, $ϵ>0$ and the stated learning rate ${\eta }_t$. The decay coefficient is $\lambda$. This explicitly fixes epsilon placement and the decay convention.^[ref]

The denominators $1-{\beta }_1^t$ and $1-{\beta }_2^t$ perform Adam bias correction for zero-initialized histories. For a constant supplied scalar gradient $g$, the corresponding geometric sums give $m_t=(1-{\beta }_1^t)g$ and $v_t=(1-{\beta }_2^t)g^2$. Their corrected values are therefore $g$ and $g^2$. The squared-gradient quantity concerns a raw second moment, rather than a centered variance: a constant gradient of 2 has square 4 even though its variance is zero.

For an original first-step example, choose $f(\theta )={\theta }^2$, ${\theta }_0=1$, $\eta =0.1$, $\lambda =0.2$, ${\beta }_1=0.9$, ${\beta }_2=0.99$ and $ϵ=0.1$. ThenandThe constants are chosen for transparent arithmetic. Feeding $g_1+\lambda {\theta }_0=2.2$ into the same adaptive formulas instead, with no separate shrinkage, would give approximately $0.904348$. That coupled L2 regularization also changes the stored histories. Resuming AdamW therefore requires the relevant optimizer state and step count as well as the parameter values and settings.

Coupled L2 regularization adds a quadratic parameter penalty to the objective function and sends its gradient through the same optimizer update as the data gradient. With coefficient $\lambda \ge 0$, defineSince the gradient of the squared Euclidean norm divided by two is $\theta$, a supplied data gradient $g=\nabla f(\theta )$ becomesThe factor $1/2$ fixes the convention: a penalty written as $\lambda \Vert \theta {\Vert }_2^2$ instead contributes $2\lambda \theta$.

For plain gradient descent, this substitution givesIt is therefore equivalent to decoupled weight decay with the matched coefficient in this particular update. The equivalence need not survive coordinate-dependent processing of the gradient.^[ref]

For an original fixed-scaling example, let $\theta =(1,1{)}^T$, $g=0$, $\eta =0.1$, $\lambda =0.2$ and diagonal matrix $D=\mathrm{diag}(1,0.1)$. Applying $D$ to the whole coupled gradient givesThe same quadratic penalty produces unequal coordinate shrinkage after this scaling. Uniform decoupled weight decay would instead give $(0.98,0.98{)}^T$. The example isolates a fixed scaling operation, rather than claiming to simulate a complete adaptive gradient method.

Coupling can also change the optimizer state. For example, a momentum buffer formed from $\tilde{g}$ stores contributions from $\lambda \theta$ along with the data gradient. If an adaptive accumulator uses squared gradient entries, it receives $(g_i+\lambda {\theta }_i{)}^2$, including a cross term $2\lambda g_i{\theta }_i$. Subsequent updates can therefore differ even when the next data gradient is the same. State both the penalty convention and the optimizer's gradient-processing rule when describing an L2 setting.

Decoupled weight decay applies a parameter-shrinkage contribution separately from the optimizer's processing of the data gradient. Write $u_t$ for the processed data-gradient direction, including any algorithm-specific optimizer state. Using the coefficient convention documented for PyTorch AdamW,The shrinkage term uses the pre-update parameter and is not inserted into the gradient moments.^[ref]

This differs from coupled L2 regularization, which processes $g_t+\lambda {\theta }_{t-1}$ as a single gradient input. Decoupled weight decay was introduced for adaptive optimization in the AdamW work; the original paper also writes decay coefficients with the learning-rate factor absorbed differently, so its coefficient must be translated before comparing equations.^[ref]

An original limiting case isolates the shrinkage. Suppose the processed direction is zero at every step and $\eta ,\lambda$ are constant. Repeated substitution givesWith $\eta =0.1$, $\lambda =0.2$ and ${\theta }_0=1$, one update gives 0.98 and 50 updates give approximately 0.36417. For varying learning rates, the corresponding factor isThus the same numerical decay coefficient can have a different accumulated effect under a different learning rate schedule. In this zero-direction model, $0<\eta \lambda <1$ gives sign-preserving shrinkage; other factors have different behaviour.

The update order is also part of the definition. For $\theta =1$, $u=2$, $\eta =0.1$ and $\lambda =0.2$, the stated formula gives $0.98-0.20=0.78$. Applying shrinkage after first taking the gradient step would instead give $(1-0.20)0.98=0.784$. Both operations may be described informally as decay plus a step, but they are different recurrences. An optimization checkpoint must retain the relevant state and settings to reproduce the chosen one.

Ilya Loshchilov is a researcher in optimization and machine learning whose work with Frank Hutter includes SGDR and AdamW. His Freiburg institutional profile identifies the same author and lists the two papers; it is an alumni record, rather than evidence of a current appointment.^[ref]

Their 2017 SGDR paper introduced a learning rate schedule that decreases the rate along a cosine curve, then raises it at a warm restart while retaining the current parameters. In one cycle,Here $u$ is elapsed cycle progress and $T$ its length. With ${\eta }_{\min }=0$ and ${\eta }_{\max }=0.1$, the rates at $u=0,T/2,T$ are $0.1,0.05,0$. Raising the rate starts another exploration from the learned parameters; it does not reinitialize the model.^[ref]

Their 2019 AdamW work separates decoupled weight decay from the gradient-based update. For ordinary gradient descent, adding $\lambda \Vert \theta {\Vert }^2/2$ gives ${\theta }^{\prime }=(1-\eta \lambda )\theta -\eta g$. Coordinate-dependent scaling can act differently on the penalty contribution, so that equivalence does not extend unchanged to an adaptive gradient method.^[ref] Writing the equation makes the contribution clearer than comparing optimizer names alone.

Weight decay supplies a parameter-shrinkage contribution to an optimizer update, with a coefficient commonly denoted $\lambda$. Two implementations must be distinguished. Coupled L2 regularization adds the penalty $\lambda \Vert \theta {\Vert }^2/2$ to the objective function, replacing its data gradient $g$ by $g+\lambda \theta$. Decoupled weight decay applies parameter shrinkage separately from the optimizer's processing of the data gradient. These operations are equivalent for plain gradient descent with matched coefficients, but not in general for an adaptive gradient method.^[ref]

Use the PyTorch convention here: a decoupled update shrinks parameters by $1-\eta \lambda$, where $\eta$ is the learning rate. A source writing $1-\lambda$ instead has absorbed the step-size factor into its decay coefficient. For an unmodified gradient step,In AdamW, the decay contribution does not enter the stored first- or second-moment estimates of the gradient.^[ref]

An original fixed-scaling example exposes the difference. Let $\theta =(2,2{)}^T$, $g=(1,1{)}^T$, $\eta =0.1$, $\lambda =0.2$ and diagonal matrix $A=\mathrm{diag}(1,10)$. Applying the same scaling to the entire coupled gradient givesSeparating uniform parameter shrinkage givesThe second coordinate receives different shrinkage. This is an illustrative fixed matrix, not a full AdamW simulation. With history-dependent scaling, coupling the penalty into the gradient can also alter the optimizer state. State the update equation and coefficient convention when reporting a decay setting; the label alone does not specify either.