 Yoram Singer

Yoram Singer coauthored the 2011 AdaGrad paper with John Duchi and Elad Hazan. Their adaptive gradient methods use past subgradients to change the geometry of later optimizer updates.^[ref]

An unconstrained diagonal specialization uses a diagonal AdaGrad accumulatorwith learning rate $\eta >0$ and stabilization $\delta >0$ outside the square root. The accumulator is part of the optimizer state.^[ref]

For an original supplied-gradient example, let the current accumulator be $s_t=(100,1)$, the current gradient be $g_t=(1,1)$, and choose $\eta =0.1$, $\delta =1$. These accumulator values include the current gradient. The next displacement isThe identical current gradient coordinates produce different movements because their histories differ. Plain gradient descent with the same learning rate would move by $(-0.1,-0.1)$. This calculation exposes the mechanism without asserting that either update improves every possible objective function.

The example also explains why saving only the parameter vector is insufficient to reproduce a later adaptive step. Two runs at the same $x_t$, receiving the same $g_t$, can move differently if their stored accumulators differ.