= Diagonal AdaGrad accumulator
{title2=$s_{t,i}=\sum_{\tau=1}^t g_{\tau,i}^2$}
The \b[diagonal AdaGrad accumulator] stores a cumulative squared-gradient total for each coordinate:
$$
s_{0,i}=0,\qquad
s_{t,i}=s_{t-1,i}+g_{t,i}^2
=\sum_{\tau=1}^{t}g_{\tau,i}^2.
$$
These quantities form part of <AdaGrad>'s <optimizer state>. An unconstrained diagonal update uses them through a coordinate scale such as $1/(\sqrt{s_{t,i}}+\epsilon)$ with a declared positive stabilizer $\epsilon$. The accumulator itself and its inverse-square-root transformation are distinct objects.<Adaptive Subgradient Methods for Online Learning and Stochastic Optimization by John Duchi and others>[Section1.1, printed pp2122–2124, Eq1 and Eq5]{ref}
The relation to the full gradient history is exact. Form the sum of <outer products>
$$
G_t=\sum_{\tau=1}^{t}g_\tau g_\tau^T.
$$
Then $(G_t)_{ii}=s_{t,i}$. Retaining only this diagonal is cheaper than retaining every entry of $G_t$, but discards cross-coordinate information.
For an original two-step example, compare histories
$$
\begin{aligned}
A:\quad &g_1=(1,1)^T,\quad g_2=(1,-1)^T,\\
B:\quad &g_1=(1,1)^T,\quad g_2=(1,1)^T.
\end{aligned}
$$
Both give $s_2=(2,2)^T$. Their full matrices differ:
$$
G_2^{A}=\begin{pmatrix}2&0\\0&2\end{pmatrix},
\qquad
G_2^{B}=\begin{pmatrix}2&2\\2&2\end{pmatrix}.
$$
The diagonal state cannot distinguish those joint gradient patterns. If both are next supplied with $g_3=(1,0)^T$, both accumulators become $(3,2)^T$ and produce the same coordinate scaling for that update.
Every entry is nonnegative and nondecreasing, since each added square is nonnegative. Alternating gradient signs therefore cannot cancel their contribution to this state. For example, scalar gradients 3 and −3 give accumulated value 18 rather than 0. A saved parameter vector does not reconstruct that history: resuming the same <AdaGrad> recurrence requires the corresponding accumulator and settings in the <optimization checkpoint>.
Back to article page
