 Optimizer state

The optimizer state is information retained by an optimization algorithm beyond the current parameter values. It can include a momentum buffer and algorithm-specific counters or running estimates. Consequently, two runs with identical parameters and current gradients can produce different next optimizer updates if their stored states differ.

For an original scalar example, minimize $f(\theta )={\theta }^2/2$ with the recurrenceThis is the momentum form with dampening, decay and the nesterov flag disabled in the PyTorch stochastic gradient descent implementation.^[ref] At ${\theta }_k=1$, $\eta =0.1$ and $\mu =0.9$, a stored $b_k=0$ gives $b_{k+1}=1$ and ${\theta }_{k+1}=0.9$. A stored $b_k=2$ instead gives $b_{k+1}=2.8$ and ${\theta }_{k+1}=0.72$. The current parameter and gradient were identical; the difference came from history.

An optimization checkpoint that saves only the parameters cannot reproduce that next step. In PyTorch, Optimizer.state_dict() stores per-parameter state and parameter-group settings such as learning rate, but not the parameter values themselves. Its parameter identifiers associate state with ordered parameter groups and are not a check of semantic parameter identity during loading.^[ref] Save and restore the corresponding parameters separately and keep the state-to-parameter association consistent. Reproducing an entire training run may additionally require the data order and other changing state.