= Optimization checkpoint
An \b[optimization checkpoint] is a saved description of an ongoing <optimization algorithm> from which its <optimizer updates> can resume. It combines the current parameters with the relevant <optimizer state>. The practical criterion is <optimizer restart equivalence>: after restoring the saved information and supplying the same subsequent inputs, the chosen update rule should produce the same subsequent states.
For a deterministic scalar example, minimize the <quadratic function> $f(\theta)=\theta^2/2$ with <gradient> $g_k=\theta_k$, <learning rate> $\eta=0.1$ and <momentum buffer> recurrence $b_{k+1}=0.9b_k+g_k$, followed by $\theta_{k+1}=\theta_k-0.1b_{k+1}$. Start with $(\theta_0,b_0)=(1,0)$. After one step the saved pair is $(\theta_1,b_1)=(0.9,1)$. Restoring both gives $b_2=1.8$ and $\theta_2=0.72$. Restoring only $\theta_1$ and initializing a new <momentum buffer> to zero gives $b_2=0.9$ and $\theta_2=0.81$. \b[Equal saved parameters can therefore lead to unequal continuations.] These numbers are an original calculation for the declared recurrence.
In <PyTorch>, a general training checkpoint commonly serializes separate model and optimizer dictionaries, together with the training position. The model dictionary contains learned parameters and registered model buffers; the optimizer dictionary contains its own changing state and group settings.<Saving and Loading Models in the PyTorch tutorials>[What is a state_dict?; Saving & Loading a General Checkpoint for Inference and/or Resuming Training]{ref} Restoring the <optimizer state-to-parameter association> also matters: the optimizer loader pairs stored identifiers with parameters in group order, without independently establishing that they denote the same mathematical parameters.<torch.optim in the PyTorch documentation>[PyTorch2.15, Optimizer.state_dict, state/param_groups and loading note]{ref}
The required checkpoint boundary depends on the computation being resumed. A changing <learning rate schedule>, the next position in the <training data>, and the state of a <pseudorandom number generator> can influence future updates. Matching them is part of <computational reproducibility>; a saved parameter vector alone does not specify the whole training process.
Back to article page
