= Cosine learning rate schedule
{title2=$\eta(t)$}
A \b[cosine learning rate schedule] varies the <learning rate> along a segment of a <cosine> curve. For a single decreasing segment, declare $t$ to count completed schedule advances and choose a duration $T>0$. With no competing rate updates, set
$$
\eta(t)=\eta_{\min}+\frac{\eta_{\max}-\eta_{\min}}2\left(1+\cos\frac{\pi t}{T}\right),\qquad0\leq t\leq T.
$$
It starts at $\eta_{\max}$, reaches the midpoint of the two bounds at $T/2$, and ends at $\eta_{\min}$. This is the cosine segment used in <SGDR>; repeating such segments with restarts is an additional policy, not part of the single-segment formula.<SGDR: Stochastic Gradient Descent with Warm Restarts by Ilya Loshchilov and Frank Hutter>[§3, cosine annealing rule and restart cycles]{ref}
For an original example, let $\eta_{\max}=0.1$, $\eta_{\min}=0.01$ and $T=4$. At $t=0,1,2,3,4$ the rates are approximately $0.1,0.08682,0.055,0.02318,0.01$. Differentiating the continuous interpolation gives
$$
\eta'(t)=-\frac{\pi(\eta_{\max}-\eta_{\min})}{2T}\sin\frac{\pi t}{T}.
$$
Thus the rate decreases throughout the interior when the upper bound exceeds the lower bound, and its interpolation has zero slope at both ends. A smooth rate curve does not by itself prove convergence of the learning procedure.
The clock must be declared: advancing once per <training epoch> differs from advancing after every parameter update. The PyTorch `CosineAnnealingLR` documentation explicitly distinguishes its implementation without restarts from <SGDR>. Its example advances the scheduler after the training and validation operations. To stop at the lower bound, stop or hold the schedule at the declared endpoint; indefinitely extending the cosine formula is not a permanently decreasing rule.<CosineAnnealingLR in the PyTorch documentation>[Class description, restart note, T_max and example loop]{ref}
Back to article page
