 Optimization checkpoint

An optimization checkpoint is a saved description of an ongoing optimization algorithm from which its optimizer updates can resume. It combines the current parameters with the relevant optimizer state. The practical criterion is optimizer restart equivalence: after restoring the saved information and supplying the same subsequent inputs, the chosen update rule should produce the same subsequent states.

For a deterministic scalar example, minimize the quadratic function $f(\theta )={\theta }^2/2$ with gradient $g_k={\theta }_k$, learning rate $\eta =0.1$ and momentum buffer recurrence $b_{k+1}=0.9b_k+g_k$, followed by ${\theta }_{k+1}={\theta }_k-0.1b_{k+1}$. Start with $({\theta }_0,b_0)=(1,0)$. After one step the saved pair is $({\theta }_1,b_1)=(0.9,1)$. Restoring both gives $b_2=1.8$ and ${\theta }_2=0.72$. Restoring only ${\theta }_1$ and initializing a new momentum buffer to zero gives $b_2=0.9$ and ${\theta }_2=0.81$. Equal saved parameters can therefore lead to unequal continuations. These numbers are an original calculation for the declared recurrence.

In PyTorch, a general training checkpoint commonly serializes separate model and optimizer dictionaries, together with the training position. The model dictionary contains learned parameters and registered model buffers; the optimizer dictionary contains its own changing state and group settings.^[ref] Restoring the optimizer state-to-parameter association also matters: the optimizer loader pairs stored identifiers with parameters in group order, without independently establishing that they denote the same mathematical parameters.^[ref]

The required checkpoint boundary depends on the computation being resumed. A changing learning rate schedule, the next position in the training data, and the state of a pseudorandom number generator can influence future updates. Matching them is part of computational reproducibility; a saved parameter vector alone does not specify the whole training process.