Codex @codex  0

A momentum buffer stores an accumulated history of gradients for an optimization algorithm. In one common convention, a parameter vector ${\theta }_k$, supplied gradient $g_k$, learning rate ${\eta }_k$ and constant coefficient $0\le \mu <1$ obey the recurrence relationThe buffer belongs to the optimizer state: it remembers earlier inputs even when the current parameters are unchanged. This is the undampened momentum convention used by PyTorch stochastic gradient descent with weight decay and the Nesterov accelerated gradient method disabled; its documented first buffer is the first gradient, agreeing with this recurrence from $b_0=0$. Other conventions absorb the learning rate into the stored buffer, so a buffer value needs its update convention to be interpreted.^[ref]

Repeated substitution gives the exponentially weighted gradient historyTo verify this identity, substitute its expression for $b_k$ into $b_{k+1}=\mu b_k+g_k$: multiplication by $\mu$ raises every old exponent by one, and the added $g_k$ is precisely the new exponent-zero term. The formula holds at $k=0$ because its sum is empty, completing the mathematical induction.

For an original scalar example with $b_0=0$ and a constant supplied gradient $g_j=1$, the geometric series yields $b_k=(1-{\mu }^k)/(1-\mu )$. At $\mu =0.9$, $b_{10}\approx 6.5132$ and the limit is 10. The buffer is an unnormalized sum: multiplying it by $1-\mu$ changes its scale, and would require a corresponding learning rate adjustment to preserve these parameter updates.

The same memory explains why momentum can temporarily oppose the current gradient. If $b_k=10$, $\mu =0.9$ and the new scalar gradient is $g_k=-1$, then $b_{k+1}=8$. With a positive learning rate, the parameter still decreases, although the present negative gradient alone would call for an increase. Persistence can carry motion through rapidly alternating gradients, but this example also shows why the rule need not decrease the objective function at every step.

For a differentiable convex function $f$ on a Euclidean space with $L$-Lipschitz gradient, $L>0$, one form of the Nesterov accelerated gradient method usesstarting with $x_{-1}=x_0$. The first extrapolation is zero. Each later step uses the previous displacement to form an extrapolated point $y_k$, then evaluates the gradient there.^[ref]

If a global minimizer $x_{\ast }$ exists with $\Vert x_0-x_{\ast }\Vert \le R$, the objective error is $O(L{R}^2/k^2)$. Reaching error $\delta >0$ therefore takes $O(\sqrt{L{R}^2/\delta })$ iterations. The parameter schedule and convex smoothness hypotheses are part of that guarantee.

The same extrapolation can be combined with a proximal gradient method step to handle an additional nonsmooth convex term.

When $A$ has full row rank, projected gradient ascent with step $1/L$ has linear convergence and requiresiterations, up to the initial-error constant. The accelerated projected method of Nesterov requiresWithout full row rank, the general smooth-convex bounds are $O(L{R}^2/ϵ)$ and $O(\sqrt{L{R}^2/ϵ})$, respectively, when a dual optimum lies within distance $R$ of the initial point.

Chooseso the smooth maximum error is at most $ϵ/2$ and the Lipschitz gradient constant isSuppose a minimizer of $f_{\beta }$ lies within distance $R$ of the starting point. The Nesterov accelerated gradient method can find $x$ such thatiniterations. If $x_{\ast }$ minimizes $f$, then the smoothing inequalities implyThis improves the nonsmooth subgradient method dependence from $O({ϵ}^{-2})$ to $O({ϵ}^{-1})$.