Adaptive gradient method

An adaptive gradient method uses information from supplied gradients to adjust subsequent optimizer updates, often with a different scale for each coordinate. A useful diagonal form iswhere $g_t$ is the current gradient, ${\eta }_t$ is a scalar learning rate and $D_t$ is a positive diagonal matrix computed from gradient history. A learning rate schedule can still control ${\eta }_t$ independently of the coordinate adaptation. The broader family includes AdaGrad, Adam and AdamW.

The 2011 AdaGrad paper by John Duchi, Elad Hazan and Yoram Singer formalized adaptive geometry from accumulated gradient information, including diagonal and full-matrix constructions.^[ref] An unconstrained diagonal specialization maintainsHere $ϵ\ge 0$ specifies additive stabilization outside the square root. The accumulator is part of the optimizer state, rather than determined by the current parameters alone.

For an original example, start at ${\theta }_0=(0,0)$ and supply $g_1=(10,1)$, with $\eta =0.1$ and $ϵ=0$. Both accumulated entries are positive, so the denominators are defined. The first step givesPlain gradient descent with the same scalar learning rate would instead give $(-1,-0.1)$. If the next supplied gradient is again $(10,1)$, then $s_2=(200,2)$ and each coordinate's additional movement is $-0.1/\sqrt{2}$. These are explicit supplied-gradient calculations, rather than a convergence claim.

In this cumulative version, an accumulated squared-gradient entry never decreases, so its effective coordinate learning rate does not increase. Other adaptive gradient methods retain decaying averages instead, changing how strongly old gradients influence later updates. Adaptation therefore specifies both the state recurrence and how that state transforms the gradient; the word “adaptive” alone does not determine either.

Adam, named for adaptive moment estimation, is an adaptive gradient method that scales a running gradient estimate by a running squared-gradient estimate. Diederik P. Kingma and Jimmy Lei Ba presented it as a first-order method for stochastic optimization. Its state makes the next optimizer update depend on gradient history as well as the current parameters.^[ref]

Starting from $m_0=v_0=0$, the basic minimization rule isHere $g_t$ is the supplied gradient, ${\eta }_t>0$ is the learning rate, $0\le {\beta }_1,{\beta }_2<1$, and $ϵ>0$. The Hadamard product, square root and division act coordinatewise. Adam bias correction supplies the two normalization factors. The accumulator $v_t$ concerns a raw second moment, rather than a centered variance.^[ref]

For an original scalar calculation, choose ${\theta }_0=0$, ${\beta }_1={\beta }_2=1/2$, ${\eta }_t=0.1$ and $ϵ=0.1$, and supply $g_1=2$, $g_2=-2$. The first step hasThe second hasEqual and opposite supplied gradients have not undone one another: the running histories give the second step a different magnitude. Plain gradient descent with the same constant learning rate would return to zero for this sequence.

The calculation specifies the recurrence, rather than a convergence guarantee for an unspecified objective function. An optimization checkpoint needs these histories and the step count to reproduce subsequent steps. AdamW adds decoupled weight decay; feeding a penalty gradient into Adam instead implements coupled L2 regularization and changes the histories.

Adam bias correction compensates for the initial zero values in Adam's exponentially weighted gradient histories. The basic algorithm usesThese factors normalize the coefficient sums of the two running histories.^[ref]

For a scalar coordinate, suppose $m_0=0$ andExpanding the recurrence relation givesIf every $g_s$ has the same expectation $\mu$, linearity of expectation and the geometric series yieldThus ${\hat{m}}_t$ is an unbiased estimator of $\mu$ under these assumptions. Correlation between different times is allowed by this argument. Applying the same calculation to squared gradients with constant raw second moment $\nu =\mathbb{E}[g_s^2]$ gives $\mathbb{E}[v_t]=(1-{\beta }_2^t)\nu$ and $\mathbb{E}[{\hat{v}}_t]=\nu$.

For a constant supplied gradient of 2, choose ${\beta }_1=0.9$ and ${\beta }_2=0.99$. The first uncorrected histories are $m_1=0.2$ and $v_1=0.04$, while their corrected values are 2 and 4. The normalization removes the missing coefficient mass caused by initialization.

It does not make a changing history equal to the latest gradient. For an original deterministic example, let ${\beta }_1=1/2$, $g_1=1$ and $g_2=3$. Then $m_2=7/4$, sowhich differs from $g_2=3$. More generally, changing moment expectations produce a weighted average of past expectations. State the moment assumptions before interpreting “bias-corrected” as unbiased for a present distribution. The nonlinear parameter update built from the corrected histories is a separate object from either moment estimator.

AdaGrad is a family of adaptive gradient methods that derives update geometry from accumulated gradient information. Its diagonal form uses a separate cumulative squared-gradient history for each coordinate. John Duchi, Elad Hazan and Yoram Singer's original paper develops diagonal and full-matrix versions, including constrained and regularized settings.^[ref]

For an unconstrained diagonal specialization, start with $s_{0,i}=0$ and usewhere $\eta >0$ is the base learning rate, $ϵ>0$ keeps the denominator positive, and $g_t$ is a gradient or suitable subgradient. The additive stabilizer is outside the square root in this declared variant. The accumulator belongs to the optimizer state and must be retained to reproduce later updates.

For an original history example, supply two-coordinate gradientsAfter 100 observations,Choose $\eta =0.1$ and $ϵ=1$. The effective coordinate learning rates are thenAlthough the recent supplied gradients agree, the early large first-coordinate value still strongly reduces its update scale.

Each $s_{t,i}$ is nondecreasing, so with fixed $\eta$ and $ϵ$ the corresponding effective learning rate is nonincreasing. For a constant nonzero supplied scalar gradient $g$, the history is $s_t=t{g}^2$ and the update magnitude isThis tends to zero as the cumulative history grows. Adam instead uses a decaying squared-gradient history, giving old observations a different influence. The choice between these histories is part of the algorithm, rather than merely a different initial learning rate.

The diagonal AdaGrad accumulator stores a cumulative squared-gradient total for each coordinate:These quantities form part of AdaGrad's optimizer state. An unconstrained diagonal update uses them through a coordinate scale such as $1/(\sqrt{s_{t,i}}+ϵ)$ with a declared positive stabilizer $ϵ$. The accumulator itself and its inverse-square-root transformation are distinct objects.^[ref]

The relation to the full gradient history is exact. Form the sum of outer productsThen $(G_t{)}_{ii}=s_{t,i}$. Retaining only this diagonal is cheaper than retaining every entry of $G_t$, but discards cross-coordinate information.

For an original two-step example, compare historiesBoth give $s_2=(2,2{)}^T$. Their full matrices differ:The diagonal state cannot distinguish those joint gradient patterns. If both are next supplied with $g_3=(1,0{)}^T$, both accumulators become $(3,2{)}^T$ and produce the same coordinate scaling for that update.

Every entry is nonnegative and nondecreasing, since each added square is nonnegative. Alternating gradient signs therefore cannot cancel their contribution to this state. For example, scalar gradients 3 and −3 give accumulated value 18 rather than 0. A saved parameter vector does not reconstruct that history: resuming the same AdaGrad recurrence requires the corresponding accumulator and settings in the optimization checkpoint.

AdamW combines the adaptive update of Adam with decoupled weight decay. It maintains exponentially weighted histories of the supplied data gradient and its componentwise square, while applying parameter shrinkage separately. Ilya Loshchilov and Frank Hutter presented the method in their work on decoupling decay from gradient-based regularization.^[ref]

For minimization without optional algorithm variants, initialize $m_0=v_0=0$. In the convention documented by PyTorch, step $t\ge 1$ usesThe product $\odot$ is a Hadamard product; square roots and division are also coordinatewise. Assume $0\le {\beta }_1,{\beta }_2<1$, $ϵ>0$ and the stated learning rate ${\eta }_t$. The decay coefficient is $\lambda$. This explicitly fixes epsilon placement and the decay convention.^[ref]

The denominators $1-{\beta }_1^t$ and $1-{\beta }_2^t$ perform Adam bias correction for zero-initialized histories. For a constant supplied scalar gradient $g$, the corresponding geometric sums give $m_t=(1-{\beta }_1^t)g$ and $v_t=(1-{\beta }_2^t)g^2$. Their corrected values are therefore $g$ and $g^2$. The squared-gradient quantity concerns a raw second moment, rather than a centered variance: a constant gradient of 2 has square 4 even though its variance is zero.

For an original first-step example, choose $f(\theta )={\theta }^2$, ${\theta }_0=1$, $\eta =0.1$, $\lambda =0.2$, ${\beta }_1=0.9$, ${\beta }_2=0.99$ and $ϵ=0.1$. ThenandThe constants are chosen for transparent arithmetic. Feeding $g_1+\lambda {\theta }_0=2.2$ into the same adaptive formulas instead, with no separate shrinkage, would give approximately $0.904348$. That coupled L2 regularization also changes the stored histories. Resuming AdamW therefore requires the relevant optimizer state and step count as well as the parameter values and settings.