{c}

\b[SGDR] means <stochastic gradient descent> with warm restarts. Its <learning rate schedule> repeatedly lowers the learning rate along a cosine curve, then raises it to begin another cycle while retaining the current parameters. <Ilya Loshchilov> and <Frank Hutter>'s method uses this <warm restart (optimization)> to continue an existing optimization trajectory.<SGDR: Stochastic Gradient Descent with Warm Restarts by Ilya Loshchilov and Frank Hutter>[Version5, section3, printed p4]{ref}

In cycle $i$, let $s$ be elapsed training time since the last restart and $T_i>0$ the cycle duration, measured in the same units. The <cosine learning rate schedule> is
$$
\eta_i(s)=\eta_{\min,i}
+\frac{\eta_{\max,i}-\eta_{\min,i}}{2}
\left(1+\cos\frac{\pi s}{T_i}\right),
\qquad 0\leq s\leq T_i.
$$
It starts at $\eta_{\max,i}$, reaches the midpoint of the two bounds at $s=T_i/2$, and reaches $\eta_{\min,i}$ at the cycle endpoint. The original implementation updates the elapsed <training epoch> count within each <mini-batch>, allowing fractional epoch progress.<SGDR: Stochastic Gradient Descent with Warm Restarts by Ilya Loshchilov and Frank Hutter>[Section3, Eq5 and definition of T_cur]{ref}

For an original schedule, choose bounds 0 and 0.1, initial duration $T_0=4$ epochs and duration multiplier 2. The successive cycle lengths are 4, 8 and 16 epochs, with restart boundaries at total elapsed times 4, 12 and 28. In the first continuous cycle, the rates at $s=0,2,4$ are respectively $0.1,0.05,0$.

\Image[/sgdr-cosine-restart-cycles.png]
{title=Cosine learning-rate cycles with doubling restart periods}
{description=Own chosen schedule with bounds0 and0.1 and durations4,8,16epochs. Open markers show end-of-cycle limits; filled markers show the rate after restart. The figure plots a schedule, not measured training performance.}

A discrete convention must specify which update uses the endpoint. For example, sampling phases $s=0,1,2,3$ in the first cycle and restarting before the next step gives rates approximately $0.1,0.08536,0.05,0.01464$, followed by $0.1$. The zero endpoint in the continuous curve is then a limit, rather than an additional zero-rate optimizer step.

Retaining parameters also has a concrete meaning. In an original deterministic <gradient descent> example with $f(\theta)=\theta^2/2$ and current $\theta=0.5$, resetting only the schedule leaves $\theta=0.5$. The next step at learning rate0.1 gives0.45; a step at0.01 would give0.495. The restart changes the next movement rather than drawing a new initial parameter. Handling a <momentum buffer> is a further state convention; restarting the schedule and discarding all <optimizer state> are different operations. Record cycle phase, durations and rate bounds with an <optimization checkpoint>.


 SGDR (source code)