= Optimizer state
The <optimizer state> is information retained by an optimization algorithm beyond the current parameter values. It can include a <momentum buffer> and algorithm-specific counters or running estimates. Consequently, two runs with identical parameters and current <gradients> can produce different next <optimizer updates> if their stored states differ.
For an original scalar example, minimize $f(\theta)=\theta^2/2$ with the recurrence
$$
b_{k+1}=\mu b_k+g_k,\qquad\theta_{k+1}=\theta_k-\eta b_{k+1},\qquad g_k=\theta_k.
$$
This is the momentum form with dampening, decay and the `nesterov` flag disabled in the <PyTorch> <stochastic gradient descent> implementation.<SGD in the PyTorch documentation>[PyTorch2.15, algorithm pseudocode; momentum and dampening description]{ref} At $\theta_k=1$, $\eta=0.1$ and $\mu=0.9$, a stored $b_k=0$ gives $b_{k+1}=1$ and $\theta_{k+1}=0.9$. A stored $b_k=2$ instead gives $b_{k+1}=2.8$ and $\theta_{k+1}=0.72$. The current parameter and gradient were identical; the difference came from history.
An <optimization checkpoint> that saves only the parameters cannot reproduce that next step. In PyTorch, `Optimizer.state_dict()` stores per-parameter state and parameter-group settings such as <learning rate>, but not the parameter values themselves. Its parameter identifiers associate state with ordered parameter groups and are not a check of semantic parameter identity during loading.<torch.optim in the PyTorch documentation>[PyTorch2.15, Optimizer.state_dict API, state/param_groups entries and loading note]{ref} Save and restore the corresponding parameters separately and keep the state-to-parameter association consistent. Reproducing an entire training run may additionally require the data order and other changing state.
\Image[/optimizer-state-history.png]
{title=Original deterministic quadratic model: keeping or resetting the momentum buffer gives different trajectories from the same parameter value}
Back to article page
