= Warm restart
{disambiguate=optimization}
A \b[warm restart] continues an <optimization> process from an already obtained solution while restarting specified algorithm controls or history. Its meaning depends on which fields are retained. In <SGDR>, the current parameters are retained while the <learning rate schedule> starts another cycle; the original paper explicitly distinguishes this from starting the optimization from scratch.<SGDR: Stochastic Gradient Descent with Warm Restarts by Ilya Loshchilov and Frank Hutter>[Version5, section3, printed p4, retained current solution and restarted learning rate]{ref}
Write a complete algorithm state as $(\theta,s,c)$, where $\theta$ holds parameters, $s$ is <optimizer state> and $c$ holds controls such as the schedule phase. A restart map can retain $\theta$ while replacing $c$, or can also replace selected parts of $s$. Saying only “restart” does not specify the next <optimizer update>.
For an original scalar example, take $f(\theta)=\theta^2/2$ and the <momentum buffer> recurrence
$$
b^+=\mu b+g,\qquad
\theta^+=\theta-\eta b^+,\qquad g=\theta.
$$
At current $\theta=2$, stored $b=-4$, $\mu=1/2$ and restarted learning rate $\eta=0.1$, retaining the buffer gives
$$
b^+=\tfrac12(-4)+2=0,\qquad\theta^+=2.
$$
Retaining the same parameters but resetting the buffer to zero gives $b^+=2$ and $\theta^+=1.8$. Both runs start the next update at $\theta=2$; they differ in the history carried through the restart.
A schedule-only restart can also change the next step without changing stored parameters immediately. In a plain <gradient descent> update from $\theta=2$, increasing the learning rate from 0.01 to 0.1 changes the next value from 1.98 to 1.8. The change takes effect through the subsequent gradient step.
An <optimization checkpoint> instead records enough information to reproduce a specified continuation. A deliberate warm restart may intentionally alter some of that information. Record the retained parameter state, the new schedule phase and the policy for buffers or counters before comparing trajectories or claiming <optimizer restart equivalence>.
Back to article page
