= Optimizer update
An <optimizer update> changes the adjustable parameters of an <optimization> problem according to an algorithm's current information. Evaluating a <loss function>, computing its <gradient> and changing the parameters are distinct operations. A general update may also change an <optimizer state>, such as stored information from previous steps:
$$
(\theta_{k+1},s_{k+1})=U(\theta_k,s_k,g_k,\eta_k).
$$
Here $g_k$ is the supplied gradient and $\eta_k$ a <learning rate>. The <PyTorch> interface makes this separation explicit with gradient computation followed by `optimizer.step()`.<torch.optim in the PyTorch documentation>[PyTorch2.15, Taking an optimization step, optimizer.step() example; Optimizer base class]{ref}
For plain <gradient descent> with no extra state, $\theta_{k+1}=\theta_k-\eta g_k$. In an original scalar example, choose $f(\theta)=\theta^2/2$, $\theta_0=2$ and $\eta=0.1$. The gradient is 2, so the update gives $\theta_1=1.8$ and lowers the loss from 2 to 1.62. Recompute the gradient at the new parameter value to obtain $\theta_2=1.62$. Reusing the old gradient 2 instead would give 1.6. This explicitly locates where the algorithm obtains its information.<SGD in the PyTorch documentation>[Algorithm pseudocode with momentum and weight decay set to zero]{ref}
An update is not automatically an improvement: using $\eta=3$ for the first step above gives $\theta_1=-4$ and loss 8. With <gradient accumulation>, several gradient calculations instead contribute before a single <optimizer update>; their weighting sets the <effective batch size>. Track the update count separately from the number of processed groups, particularly when applying a <learning rate schedule>.
Back to article page
