 Diagonal AdaGrad accumulator

The diagonal AdaGrad accumulator stores a cumulative squared-gradient total for each coordinate:These quantities form part of AdaGrad's optimizer state. An unconstrained diagonal update uses them through a coordinate scale such as $1/(\sqrt{s_{t,i}}+ϵ)$ with a declared positive stabilizer $ϵ$. The accumulator itself and its inverse-square-root transformation are distinct objects.^[ref]

The relation to the full gradient history is exact. Form the sum of outer productsThen $(G_t{)}_{ii}=s_{t,i}$. Retaining only this diagonal is cheaper than retaining every entry of $G_t$, but discards cross-coordinate information.

For an original two-step example, compare historiesBoth give $s_2=(2,2{)}^T$. Their full matrices differ:The diagonal state cannot distinguish those joint gradient patterns. If both are next supplied with $g_3=(1,0{)}^T$, both accumulators become $(3,2{)}^T$ and produce the same coordinate scaling for that update.

Every entry is nonnegative and nondecreasing, since each added square is nonnegative. Alternating gradient signs therefore cannot cancel their contribution to this state. For example, scalar gradients 3 and −3 give accumulated value 18 rather than 0. A saved parameter vector does not reconstruct that history: resuming the same AdaGrad recurrence requires the corresponding accumulator and settings in the optimization checkpoint.