{c}

\b[AdaGrad] is a family of <adaptive gradient methods> that derives update geometry from accumulated gradient information. Its diagonal form uses a separate cumulative squared-gradient history for each coordinate. <John Duchi>, <Elad Hazan> and <Yoram Singer>'s original paper develops diagonal and full-matrix versions, including constrained and regularized settings.<Adaptive Subgradient Methods for Online Learning and Stochastic Optimization by John Duchi and others>[Section1.1, printed pp2122–2124, Eq1 and Eq5]{ref}

For an unconstrained diagonal specialization, start with $s_{0,i}=0$ and use
$$
s_{t,i}=s_{t-1,i}+g_{t,i}^2,
\qquad
\theta_{t,i}=\theta_{t-1,i}
-\eta\frac{g_{t,i}}{\sqrt{s_{t,i}}+\epsilon},
$$
where $\eta>0$ is the base <learning rate>, $\epsilon>0$ keeps the denominator positive, and $g_t$ is a <gradient> or suitable <subgradient>. The additive stabilizer is outside the square root in this declared variant. The accumulator belongs to the <optimizer state> and must be retained to reproduce later updates.

For an original history example, supply two-coordinate gradients
$$
g_1=(1000,1),\qquad
g_t=(1,1)\quad(2\leq t\leq100).
$$
After 100 observations,
$$
s_{100}=(1000^2+99,\ 100)=(1000099,100).
$$
Choose $\eta=0.1$ and $\epsilon=1$. The effective coordinate learning rates are then
$$
\frac{0.1}{\sqrt{1000099}+1}\approx9.99\times10^{-5},
\qquad
\frac{0.1}{10+1}\approx9.09\times10^{-3}.
$$
Although the recent supplied gradients agree, the early large first-coordinate value still strongly reduces its update scale.

Each $s_{t,i}$ is nondecreasing, so with fixed $\eta$ and $\epsilon$ the corresponding effective learning rate is nonincreasing. For a constant nonzero supplied scalar gradient $g$, the history is $s_t=tg^2$ and the update magnitude is
$$
\frac{\eta|g|}{|g|\sqrt t+\epsilon}.
$$
This tends to zero as the cumulative history grows. <Adam> instead uses a decaying squared-gradient history, giving old observations a different influence. The choice between these histories is part of the algorithm, rather than merely a different initial learning rate.


 AdaGrad (source code)