AdaGrad

AdaGrad is a family of adaptive gradient methods that derives update geometry from accumulated gradient information. Its diagonal form uses a separate cumulative squared-gradient history for each coordinate. John Duchi, Elad Hazan and Yoram Singer's original paper develops diagonal and full-matrix versions, including constrained and regularized settings.^[ref]

For an unconstrained diagonal specialization, start with $s_{0,i}=0$ and usewhere $\eta >0$ is the base learning rate, $ϵ>0$ keeps the denominator positive, and $g_t$ is a gradient or suitable subgradient. The additive stabilizer is outside the square root in this declared variant. The accumulator belongs to the optimizer state and must be retained to reproduce later updates.

For an original history example, supply two-coordinate gradientsAfter 100 observations,Choose $\eta =0.1$ and $ϵ=1$. The effective coordinate learning rates are thenAlthough the recent supplied gradients agree, the early large first-coordinate value still strongly reduces its update scale.

Each $s_{t,i}$ is nondecreasing, so with fixed $\eta$ and $ϵ$ the corresponding effective learning rate is nonincreasing. For a constant nonzero supplied scalar gradient $g$, the history is $s_t=t{g}^2$ and the update magnitude isThis tends to zero as the cumulative history grows. Adam instead uses a decaying squared-gradient history, giving old observations a different influence. The choice between these histories is part of the algorithm, rather than merely a different initial learning rate.

The diagonal AdaGrad accumulator stores a cumulative squared-gradient total for each coordinate:These quantities form part of AdaGrad's optimizer state. An unconstrained diagonal update uses them through a coordinate scale such as $1/(\sqrt{s_{t,i}}+ϵ)$ with a declared positive stabilizer $ϵ$. The accumulator itself and its inverse-square-root transformation are distinct objects.^[ref]

The relation to the full gradient history is exact. Form the sum of outer productsThen $(G_t{)}_{ii}=s_{t,i}$. Retaining only this diagonal is cheaper than retaining every entry of $G_t$, but discards cross-coordinate information.

For an original two-step example, compare historiesBoth give $s_2=(2,2{)}^T$. Their full matrices differ:The diagonal state cannot distinguish those joint gradient patterns. If both are next supplied with $g_3=(1,0{)}^T$, both accumulators become $(3,2{)}^T$ and produce the same coordinate scaling for that update.

Every entry is nonnegative and nondecreasing, since each added square is nonnegative. Alternating gradient signs therefore cannot cancel their contribution to this state. For example, scalar gradients 3 and −3 give accumulated value 18 rather than 0. A saved parameter vector does not reconstruct that history: resuming the same AdaGrad recurrence requires the corresponding accumulator and settings in the optimization checkpoint.