 AdaGrad

AdaGrad is a family of adaptive gradient methods that derives update geometry from accumulated gradient information. Its diagonal form uses a separate cumulative squared-gradient history for each coordinate. John Duchi, Elad Hazan and Yoram Singer's original paper develops diagonal and full-matrix versions, including constrained and regularized settings.^[ref]

For an unconstrained diagonal specialization, start with $s_{0,i}=0$ and usewhere $\eta >0$ is the base learning rate, $ϵ>0$ keeps the denominator positive, and $g_t$ is a gradient or suitable subgradient. The additive stabilizer is outside the square root in this declared variant. The accumulator belongs to the optimizer state and must be retained to reproduce later updates.

For an original history example, supply two-coordinate gradientsAfter 100 observations,Choose $\eta =0.1$ and $ϵ=1$. The effective coordinate learning rates are thenAlthough the recent supplied gradients agree, the early large first-coordinate value still strongly reduces its update scale.

Each $s_{t,i}$ is nondecreasing, so with fixed $\eta$ and $ϵ$ the corresponding effective learning rate is nonincreasing. For a constant nonzero supplied scalar gradient $g$, the history is $s_t=t{g}^2$ and the update magnitude isThis tends to zero as the cumulative history grows. Adam instead uses a decaying squared-gradient history, giving old observations a different influence. The choice between these histories is part of the algorithm, rather than merely a different initial learning rate.