 Optimizer update

An optimizer update changes the adjustable parameters of an optimization problem according to an algorithm's current information. Evaluating a loss function, computing its gradient and changing the parameters are distinct operations. A general update may also change an optimizer state, such as stored information from previous steps:Here $g_k$ is the supplied gradient and ${\eta }_k$ a learning rate. The PyTorch interface makes this separation explicit with gradient computation followed by optimizer.step().^[ref]

For plain gradient descent with no extra state, ${\theta }_{k+1}={\theta }_k-\eta g_k$. In an original scalar example, choose $f(\theta )={\theta }^2/2$, ${\theta }_0=2$ and $\eta =0.1$. The gradient is 2, so the update gives ${\theta }_1=1.8$ and lowers the loss from 2 to 1.62. Recompute the gradient at the new parameter value to obtain ${\theta }_2=1.62$. Reusing the old gradient 2 instead would give 1.6. This explicitly locates where the algorithm obtains its information.^[ref]

An update is not automatically an improvement: using $\eta =3$ for the first step above gives ${\theta }_1=-4$ and loss 8. With gradient accumulation, several gradient calculations instead contribute before a single optimizer update; their weighting sets the effective batch size. Track the update count separately from the number of processed groups, particularly when applying a learning rate schedule.