= Momentum buffer
{title2=$b_k$}
A \b[momentum buffer] stores an accumulated history of <gradients> for an <optimization algorithm>. In one common convention, a parameter vector $\theta_k$, supplied <gradient> $g_k$, <learning rate> $\eta_k$ and constant coefficient $0\leq\mu<1$ obey the <recurrence relation>
$$
b_{k+1}=\mu b_k+g_k,\qquad \theta_{k+1}=\theta_k-\eta_k b_{k+1}.
$$
The buffer belongs to the <optimizer state>: it remembers earlier inputs even when the current parameters are unchanged. This is the undampened momentum convention used by <PyTorch> <stochastic gradient descent> with <weight decay> and the <Nesterov accelerated gradient method> disabled; its documented first buffer is the first gradient, agreeing with this recurrence from $b_0=0$. Other conventions absorb the <learning rate> into the stored buffer, so a buffer value needs its update convention to be interpreted.<SGD in the PyTorch documentation>[PyTorch2.15, algorithm and note comparing momentum conventions and first-step initialization]{ref}
Repeated substitution gives the <exponentially weighted gradient history>
$$
\boxed{b_k=\mu^k b_0+\sum_{j=0}^{k-1}\mu^{k-1-j}g_j}.
$$
To verify this identity, substitute its expression for $b_k$ into $b_{k+1}=\mu b_k+g_k$: multiplication by $\mu$ raises every old exponent by one, and the added $g_k$ is precisely the new exponent-zero term. The formula holds at $k=0$ because its sum is empty, completing the <mathematical induction>.
For an original scalar example with $b_0=0$ and a constant supplied <gradient> $g_j=1$, the <geometric series> yields $b_k=(1-\mu^k)/(1-\mu)$. At $\mu=0.9$, $b_{10}\approx6.5132$ and the <limit> is 10. The buffer is an unnormalized sum: multiplying it by $1-\mu$ changes its scale, and would require a corresponding <learning rate> adjustment to preserve these parameter updates.
The same memory explains why <momentum can temporarily oppose the current gradient>. If $b_k=10$, $\mu=0.9$ and the new scalar <gradient> is $g_k=-1$, then $b_{k+1}=8$. With a positive <learning rate>, the parameter still decreases, although the present negative <gradient> alone would call for an increase. Persistence can carry motion through rapidly alternating <gradients>, but this example also shows why the rule need not decrease the <objective function> at every step.
Back to article page
