= Yoram Singer
{c}
Yoram Singer coauthored the 2011 <AdaGrad> paper with <John Duchi> and <Elad Hazan>. Their <adaptive gradient methods> use past <subgradients> to change the geometry of later <optimizer updates>.<Adaptive Subgradient Methods for Online Learning and Stochastic Optimization by John Duchi and others>[Full byline; Section 1.1, equations 1 and 5]{ref}
An unconstrained diagonal specialization uses a <diagonal AdaGrad accumulator>
$$
s_{t,i}=\sum_{\tau=1}^{t}g_{\tau,i}^2,
\qquad
x_{t+1,i}=x_{t,i}-\eta\frac{g_{t,i}}{\sqrt{s_{t,i}}+\delta},
$$
with <learning rate> $\eta>0$ and stabilization $\delta>0$ outside the square root. The accumulator is part of the <optimizer state>.<Adaptive Subgradient Methods for Online Learning and Stochastic Optimization by John Duchi and others>[Section 1.1, diagonal metric in equation 5, with no constraint or extra regularizer]{ref}
For an original supplied-gradient example, let the current accumulator be $s_t=(100,1)$, the current <gradient> be $g_t=(1,1)$, and choose $\eta=0.1$, $\delta=1$. These accumulator values include the current <gradient>. The next displacement is
$$
x_{t+1}-x_t=\left(-\frac{0.1}{11},-\frac{0.1}{2}\right)
=\left(-\frac1{110},-\frac1{20}\right).
$$
The identical current <gradient> coordinates produce different movements because their histories differ. Plain <gradient descent> with the same <learning rate> would move by $(-0.1,-0.1)$. This calculation exposes the mechanism without asserting that either update improves every possible <objective function>.
The example also explains why saving only the parameter vector is insufficient to reproduce a later adaptive step. Two runs at the same $x_t$, receiving the same $g_t$, can move differently if their stored accumulators differ.
\Image[https://yoram-singer.github.io/figs/y.jpg]
{title=Yoram Singer, portrait linked from his personal research homepage}
{description=<Yoram Singer personal research homepage>[Named homepage and linked portrait]{ref}}
Back to article page
