Optimizer restart equivalence

Optimizer restart equivalence means that restoring an optimization checkpoint reproduces the uninterrupted optimizer's parameter sequence for a specified subsequent computation. State the optimizer update rule, continuation inputs and intended notion of equality. Exact mathematical equality and agreement within a numerical tolerance are different claims.

For a deterministic algorithm modeled by $S_{k+1}=F(S_k,u_k)$, equality of the restored full state at step $K$, together with the same update rule and future inputs $u_K,u_{K+1},\dots$, is sufficient. One application of $F$ gives equal next states; mathematical induction then gives equal continuations. This is a sufficient condition for the chosen observations, not a claim that every internal field must always affect them.

For an original exact example, take $f(\theta )={\theta }^2/2$ andFrom $({\theta }_0,b_0)=(2,0)$, one step gives $({\theta }_1,b_1)=(3/2,2)$. Restoring that pair gives $b_2=5/2$ and ${\theta }_2=7/8$, exactly as uninterrupted execution. Resetting the buffer while restoring ${\theta }_1$ gives $b_2=3/2$ and ${\theta }_2=9/8$. Thus a parameter-only save is not restart-equivalent for this recurrence.

Practical checkpoints must also retain the optimizer state-to-parameter association and the relevant settings, update position, learning rate schedule, data position and pseudorandom number generator state.^[ref] Fixing a random seed alone does not specify all of this continuation. Floating-point arithmetic and nondeterministic operations can additionally affect exact reproduction across environments; the primary documentation does not guarantee identical results across releases or platforms.^[ref]