= Adaptive gradient method
An \b[adaptive gradient method] uses information from supplied <gradients> to adjust subsequent <optimizer updates>, often with a different scale for each coordinate. A useful diagonal form is
$$
\theta_t=\theta_{t-1}-\eta_tD_tg_t,
$$
where $g_t$ is the current gradient, $\eta_t$ is a scalar <learning rate> and $D_t$ is a positive <diagonal matrix> computed from gradient history. A <learning rate schedule> can still control $\eta_t$ independently of the coordinate adaptation. The broader family includes <AdaGrad>, <Adam> and <AdamW>.
The 2011 <AdaGrad> paper by <John Duchi>, <Elad Hazan> and <Yoram Singer> formalized adaptive geometry from accumulated gradient information, including diagonal and full-matrix constructions.<Adaptive Subgradient Methods for Online Learning and Stochastic Optimization by John Duchi and others>[Section1.1, printed pp2122–2124, Eq1 and Eq5]{ref} An unconstrained diagonal specialization maintains
$$
s_{t,i}=\sum_{\tau=1}^{t}g_{\tau,i}^2,
\qquad
\theta_{t,i}=\theta_{t-1,i}
-\eta\frac{g_{t,i}}{\sqrt{s_{t,i}}+\epsilon}.
$$
Here $\epsilon\geq0$ specifies additive stabilization outside the square root. The accumulator is part of the <optimizer state>, rather than determined by the current parameters alone.
For an original example, start at $\theta_0=(0,0)$ and supply $g_1=(10,1)$, with $\eta=0.1$ and $\epsilon=0$. Both accumulated entries are positive, so the denominators are defined. The first step gives
$$
s_1=(100,1),\qquad \theta_1=(-0.1,-0.1).
$$
Plain <gradient descent> with the same scalar learning rate would instead give $(-1,-0.1)$. If the next supplied gradient is again $(10,1)$, then $s_2=(200,2)$ and each coordinate's additional movement is $-0.1/\sqrt2$. These are explicit supplied-gradient calculations, rather than a convergence claim.
In this cumulative version, an accumulated squared-gradient entry never decreases, so its effective coordinate learning rate does not increase. Other <adaptive gradient methods> retain decaying averages instead, changing how strongly old gradients influence later updates. Adaptation therefore specifies both the state recurrence and how that state transforms the gradient; the word “adaptive” alone does not determine either.
Back to article page
