= Exponentially weighted gradient history
{title2=$b_k=\mu^k b_0+\sum_{j=0}^{k-1}\mu^{k-1-j}g_j$}
An <exponentially weighted gradient history> assigns geometrically decreasing coefficients to older supplied <gradients>. For constant $0\le\mu<1$ and <momentum buffer> recurrence $b_{k+1}=\mu b_k+g_k$,
$$
\boxed{b_k=\mu^k b_0+\sum_{j=0}^{k-1}\mu^{k-1-j}g_j.}
$$
Each additional update multiplies an old contribution by $\mu$. The formula follows by substituting the expression for $b_k$ into the recurrence and adding the newest exponent-zero term. The same undampened recurrence, including its initialization convention, appears in the primary <stochastic gradient descent> documentation.<SGD in the PyTorch documentation>[PyTorch2.15, momentum algorithm and first-buffer initialization note]{ref}
For an original example, choose $b_0=0$, $\mu=1/2$ and gradients $g_0=2$, $g_1=-2$, $g_2=4$. Successive buffers are 2, −1, 3.5. Directly expanding the third buffer gives
$$
b_3=\tfrac14(2)+\tfrac12(-2)+4=3.5.
$$
The newest contribution has coefficient 1; the oldest has coefficient 1/4. The buffer is a weighted sum, not yet a <weighted mean>. At $b_0=0$, its coefficient sum is the <geometric series>
$$
W_k=\sum_{j=0}^{k-1}\mu^j=\frac{1-\mu^k}{1-\mu}.
$$
Here $W_3=1.75$, so the normalized <weighted mean> is $b_3/W_3=2$, rather than the ordinary mean $4/3$.
The coefficients also give a concrete memory scale. In the infinite geometric sequence, the fraction assigned to the newest $m$ gradients is $1-\mu^m$. At $\mu=0.9$, 29 gradients account for at least 95% of that coefficient mass, because $0.9^{29}<0.05$ whereas $0.9^{28}>0.05$. This is a statement about coefficients, not a guarantee that gradients at different times are independent or that the optimizer averages a stationary process.
Back to article page
