 Warm restart (optimization)

A warm restart continues an optimization process from an already obtained solution while restarting specified algorithm controls or history. Its meaning depends on which fields are retained. In SGDR, the current parameters are retained while the learning rate schedule starts another cycle; the original paper explicitly distinguishes this from starting the optimization from scratch.^[ref]

Write a complete algorithm state as $(\theta ,s,c)$, where $\theta$ holds parameters, $s$ is optimizer state and $c$ holds controls such as the schedule phase. A restart map can retain $\theta$ while replacing $c$, or can also replace selected parts of $s$. Saying only “restart” does not specify the next optimizer update.

For an original scalar example, take $f(\theta )={\theta }^2/2$ and the momentum buffer recurrenceAt current $\theta =2$, stored $b=-4$, $\mu =1/2$ and restarted learning rate $\eta =0.1$, retaining the buffer givesRetaining the same parameters but resetting the buffer to zero gives $b^{+}=2$ and ${\theta }^{+}=1.8$. Both runs start the next update at $\theta =2$; they differ in the history carried through the restart.

A schedule-only restart can also change the next step without changing stored parameters immediately. In a plain gradient descent update from $\theta =2$, increasing the learning rate from 0.01 to 0.1 changes the next value from 1.98 to 1.8. The change takes effect through the subsequent gradient step.

An optimization checkpoint instead records enough information to reproduce a specified continuation. A deliberate warm restart may intentionally alter some of that information. Record the retained parameter state, the new schedule phase and the policy for buffers or counters before comparing trajectories or claiming optimizer restart equivalence.