= Momentum can temporarily oppose the current gradient
A <momentum buffer> can retain enough earlier history that it points against the current <gradient>. For the undampened rule
$$
b_{k+1}=\mu b_k+g_k,\qquad d_k=-b_{k+1},
$$
the parameter step is $\eta d_k$ for positive <learning rate> $\eta$. If $g_k^Tb_{k+1}<0$, then the <directional derivative> in direction $d_k$ is $g_k^Td_k>0$. The actual update direction is therefore locally uphill for a differentiable <objective function>, even though gradient descent would use the opposite direction. This history-dependent recurrence is the non-Nesterov momentum convention documented for <stochastic gradient descent> with dampening and <weight decay> disabled.<SGD in the PyTorch documentation>[PyTorch2.15, algorithm and momentum-convention note]{ref}
For an original scalar example, let $f(\theta)=\theta^2/2$, current $\theta=1$, stored buffer $b=-3$, coefficient $\mu=1/2$ and $\eta=0.1$. The new gradient is 1, but
$$
b'=-\tfrac32+1=-\tfrac12,\qquad\theta'=1-0.1(-\tfrac12)=1.05.
$$
The objective increases from $f(1)=0.5$ to $f(1.05)=0.55125$. This is a fully specified finite step, not only a first-order prediction.
More generally,
$$
g_k^Tb_{k+1}=\mu g_k^Tb_k+\lVert g_k\rVert^2.
$$
An old buffer sufficiently opposed to the new gradient makes this negative. At $\mu=0$, that mechanism disappears. The example explains why a momentum method need not decrease the objective at every step; it does not imply that retaining gradient history always harms convergence. A <descent direction> test concerns the direction at the current point, while a convergence claim concerns an entire sequence.
Back to article page
