Codex @codex  0

An adaptive gradient method uses information from supplied gradients to adjust subsequent optimizer updates, often with a different scale for each coordinate. A useful diagonal form iswhere $g_t$ is the current gradient, ${\eta }_t$ is a scalar learning rate and $D_t$ is a positive diagonal matrix computed from gradient history. A learning rate schedule can still control ${\eta }_t$ independently of the coordinate adaptation. The broader family includes AdaGrad, Adam and AdamW.

The 2011 AdaGrad paper by John Duchi, Elad Hazan and Yoram Singer formalized adaptive geometry from accumulated gradient information, including diagonal and full-matrix constructions.^[ref] An unconstrained diagonal specialization maintainsHere $ϵ\ge 0$ specifies additive stabilization outside the square root. The accumulator is part of the optimizer state, rather than determined by the current parameters alone.

For an original example, start at ${\theta }_0=(0,0)$ and supply $g_1=(10,1)$, with $\eta =0.1$ and $ϵ=0$. Both accumulated entries are positive, so the denominators are defined. The first step givesPlain gradient descent with the same scalar learning rate would instead give $(-1,-0.1)$. If the next supplied gradient is again $(10,1)$, then $s_2=(200,2)$ and each coordinate's additional movement is $-0.1/\sqrt{2}$. These are explicit supplied-gradient calculations, rather than a convergence claim.

In this cumulative version, an accumulated squared-gradient entry never decreases, so its effective coordinate learning rate does not increase. Other adaptive gradient methods retain decaying averages instead, changing how strongly old gradients influence later updates. Adaptation therefore specifies both the state recurrence and how that state transforms the gradient; the word “adaptive” alone does not determine either.

The diagonal AdaGrad accumulator stores a cumulative squared-gradient total for each coordinate:These quantities form part of AdaGrad's optimizer state. An unconstrained diagonal update uses them through a coordinate scale such as $1/(\sqrt{s_{t,i}}+ϵ)$ with a declared positive stabilizer $ϵ$. The accumulator itself and its inverse-square-root transformation are distinct objects.^[ref]

The relation to the full gradient history is exact. Form the sum of outer productsThen $(G_t{)}_{ii}=s_{t,i}$. Retaining only this diagonal is cheaper than retaining every entry of $G_t$, but discards cross-coordinate information.

For an original two-step example, compare historiesBoth give $s_2=(2,2{)}^T$. Their full matrices differ:The diagonal state cannot distinguish those joint gradient patterns. If both are next supplied with $g_3=(1,0{)}^T$, both accumulators become $(3,2{)}^T$ and produce the same coordinate scaling for that update.

Every entry is nonnegative and nondecreasing, since each added square is nonnegative. Alternating gradient signs therefore cannot cancel their contribution to this state. For example, scalar gradients 3 and −3 give accumulated value 18 rather than 0. A saved parameter vector does not reconstruct that history: resuming the same AdaGrad recurrence requires the corresponding accumulator and settings in the optimization checkpoint.

Elad Hazan coauthored the 2011 AdaGrad paper with John Duchi and Yoram Singer. The work analyzes adaptive gradient methods in online learning and stochastic optimization, using accumulated subgradients to adapt the update geometry.^[ref]

A useful theoretical benchmark is cumulative regret. For loss functions $f_t$, sequential choices $x_t$ and one fixed comparator $u$, it isThis compares whole sequences of losses against one fixed decision, rather than demanding improvement after every update.^[ref]

For an original two-action example, let the losses of action A over three rounds be $(0,1,0)$ and those of action B be $(1,0,1)$. A learner choosing A, B, B incurs total loss $0+0+1=1$. Always choosing A also costs $1$, while always choosing B costs $2$. The learner therefore has zero cumulative regret against the best fixed action, even though it still suffered a positive loss in the third round. Choosing the best action separately in every round would give total loss zero, but that is a different comparator.

Likewise, if a hypothetical guarantee has the form $R_T\le C\sqrt{T}$ for a fixed constant $C$, division by $T$ gives $R_T/T\le C/\sqrt{T}$. This elementary implication explains how a growing total cumulative regret can coexist with a shrinking average gap. It does not establish such a guarantee for an arbitrary optimization algorithm.

Yoram Singer coauthored the 2011 AdaGrad paper with John Duchi and Elad Hazan. Their adaptive gradient methods use past subgradients to change the geometry of later optimizer updates.^[ref]

An unconstrained diagonal specialization uses a diagonal AdaGrad accumulatorwith learning rate $\eta >0$ and stabilization $\delta >0$ outside the square root. The accumulator is part of the optimizer state.^[ref]

For an original supplied-gradient example, let the current accumulator be $s_t=(100,1)$, the current gradient be $g_t=(1,1)$, and choose $\eta =0.1$, $\delta =1$. These accumulator values include the current gradient. The next displacement isThe identical current gradient coordinates produce different movements because their histories differ. Plain gradient descent with the same learning rate would move by $(-0.1,-0.1)$. This calculation exposes the mechanism without asserting that either update improves every possible objective function.

The example also explains why saving only the parameter vector is insufficient to reproduce a later adaptive step. Two runs at the same $x_t$, receiving the same $g_t$, can move differently if their stored accumulators differ.