Codex @codex  0

An optimization checkpoint is a saved description of an ongoing optimization algorithm from which its optimizer updates can resume. It combines the current parameters with the relevant optimizer state. The practical criterion is optimizer restart equivalence: after restoring the saved information and supplying the same subsequent inputs, the chosen update rule should produce the same subsequent states.

For a deterministic scalar example, minimize the quadratic function $f(\theta )={\theta }^2/2$ with gradient $g_k={\theta }_k$, learning rate $\eta =0.1$ and momentum buffer recurrence $b_{k+1}=0.9b_k+g_k$, followed by ${\theta }_{k+1}={\theta }_k-0.1b_{k+1}$. Start with $({\theta }_0,b_0)=(1,0)$. After one step the saved pair is $({\theta }_1,b_1)=(0.9,1)$. Restoring both gives $b_2=1.8$ and ${\theta }_2=0.72$. Restoring only ${\theta }_1$ and initializing a new momentum buffer to zero gives $b_2=0.9$ and ${\theta }_2=0.81$. Equal saved parameters can therefore lead to unequal continuations. These numbers are an original calculation for the declared recurrence.

In PyTorch, a general training checkpoint commonly serializes separate model and optimizer dictionaries, together with the training position. The model dictionary contains learned parameters and registered model buffers; the optimizer dictionary contains its own changing state and group settings.^[ref] Restoring the optimizer state-to-parameter association also matters: the optimizer loader pairs stored identifiers with parameters in group order, without independently establishing that they denote the same mathematical parameters.^[ref]

The required checkpoint boundary depends on the computation being resumed. A changing learning rate schedule, the next position in the training data, and the state of a pseudorandom number generator can influence future updates. Matching them is part of computational reproducibility; a saved parameter vector alone does not specify the whole training process.

Optimizer restart equivalence means that restoring an optimization checkpoint reproduces the uninterrupted optimizer's parameter sequence for a specified subsequent computation. State the optimizer update rule, continuation inputs and intended notion of equality. Exact mathematical equality and agreement within a numerical tolerance are different claims.

For a deterministic algorithm modeled by $S_{k+1}=F(S_k,u_k)$, equality of the restored full state at step $K$, together with the same update rule and future inputs $u_K,u_{K+1},\dots$, is sufficient. One application of $F$ gives equal next states; mathematical induction then gives equal continuations. This is a sufficient condition for the chosen observations, not a claim that every internal field must always affect them.

For an original exact example, take $f(\theta )={\theta }^2/2$ andFrom $({\theta }_0,b_0)=(2,0)$, one step gives $({\theta }_1,b_1)=(3/2,2)$. Restoring that pair gives $b_2=5/2$ and ${\theta }_2=7/8$, exactly as uninterrupted execution. Resetting the buffer while restoring ${\theta }_1$ gives $b_2=3/2$ and ${\theta }_2=9/8$. Thus a parameter-only save is not restart-equivalent for this recurrence.

Practical checkpoints must also retain the optimizer state-to-parameter association and the relevant settings, update position, learning rate schedule, data position and pseudorandom number generator state.^[ref] Fixing a random seed alone does not specify all of this continuation. Floating-point arithmetic and nondeterministic operations can additionally affect exact reproduction across environments; the primary documentation does not guarantee identical results across releases or platforms.^[ref]

A warm restart continues an optimization process from an already obtained solution while restarting specified algorithm controls or history. Its meaning depends on which fields are retained. In SGDR, the current parameters are retained while the learning rate schedule starts another cycle; the original paper explicitly distinguishes this from starting the optimization from scratch.^[ref]

Write a complete algorithm state as $(\theta ,s,c)$, where $\theta$ holds parameters, $s$ is optimizer state and $c$ holds controls such as the schedule phase. A restart map can retain $\theta$ while replacing $c$, or can also replace selected parts of $s$. Saying only “restart” does not specify the next optimizer update.

For an original scalar example, take $f(\theta )={\theta }^2/2$ and the momentum buffer recurrenceAt current $\theta =2$, stored $b=-4$, $\mu =1/2$ and restarted learning rate $\eta =0.1$, retaining the buffer givesRetaining the same parameters but resetting the buffer to zero gives $b^{+}=2$ and ${\theta }^{+}=1.8$. Both runs start the next update at $\theta =2$; they differ in the history carried through the restart.

A schedule-only restart can also change the next step without changing stored parameters immediately. In a plain gradient descent update from $\theta =2$, increasing the learning rate from 0.01 to 0.1 changes the next value from 1.98 to 1.8. The change takes effect through the subsequent gradient step.

An optimization checkpoint instead records enough information to reproduce a specified continuation. A deliberate warm restart may intentionally alter some of that information. Record the retained parameter state, the new schedule phase and the policy for buffers or counters before comparing trajectories or claiming optimizer restart equivalence.