= Optimizer restart equivalence
<Optimizer restart equivalence> means that restoring an <optimization checkpoint> reproduces the uninterrupted optimizer's parameter sequence for a specified subsequent computation. State the <optimizer update> rule, continuation inputs and intended notion of equality. Exact mathematical equality and agreement within a numerical tolerance are different claims.
For a <deterministic algorithm> modeled by $S_{k+1}=F(S_k,u_k)$, equality of the restored full state at step $K$, together with the same update rule and future inputs $u_K,u_{K+1},\ldots$, is sufficient. One application of $F$ gives equal next states; <mathematical induction> then gives equal continuations. This is a sufficient condition for the chosen observations, not a claim that every internal field must always affect them.
For an original exact example, take $f(\theta)=\theta^2/2$ and
$$
b_{k+1}=\tfrac12b_k+\theta_k,\qquad \theta_{k+1}=\theta_k-\tfrac14b_{k+1}.
$$
From $(\theta_0,b_0)=(2,0)$, one step gives $(\theta_1,b_1)=(3/2,2)$. Restoring that pair gives $b_2=5/2$ and $\theta_2=7/8$, exactly as uninterrupted execution. Resetting the buffer while restoring $\theta_1$ gives $b_2=3/2$ and $\theta_2=9/8$. Thus a parameter-only save is not restart-equivalent for this recurrence.
Practical checkpoints must also retain the <optimizer state-to-parameter association> and the relevant settings, update position, <learning rate schedule>, data position and <pseudorandom number generator> state.<Saving and Loading Models in the PyTorch tutorials>[General checkpoint for resuming training: model and optimizer dictionaries and training position]{ref} Fixing a <random seed> alone does not specify all of this continuation. <Floating-point arithmetic> and nondeterministic operations can additionally affect exact reproduction across environments; the primary documentation does not guarantee identical results across releases or platforms.<Reproducibility in the PyTorch documentation>[PyTorch2.15, opening reproducibility statement and controlling randomness]{ref}
Back to article page
