Cosine learning rate schedule ($\eta (t)$)

A cosine learning rate schedule varies the learning rate along a segment of a cosine curve. For a single decreasing segment, declare $t$ to count completed schedule advances and choose a duration $T>0$. With no competing rate updates, setIt starts at ${\eta }_{\max }$, reaches the midpoint of the two bounds at $T/2$, and ends at ${\eta }_{\min }$. This is the cosine segment used in SGDR; repeating such segments with restarts is an additional policy, not part of the single-segment formula.^[ref]

For an original example, let ${\eta }_{\max }=0.1$, ${\eta }_{\min }=0.01$ and $T=4$. At $t=0,1,2,3,4$ the rates are approximately $0.1,0.08682,0.055,0.02318,0.01$. Differentiating the continuous interpolation givesThus the rate decreases throughout the interior when the upper bound exceeds the lower bound, and its interpolation has zero slope at both ends. A smooth rate curve does not by itself prove convergence of the learning procedure.

The clock must be declared: advancing once per training epoch differs from advancing after every parameter update. The PyTorch CosineAnnealingLR documentation explicitly distinguishes its implementation without restarts from SGDR. Its example advances the scheduler after the training and validation operations. To stop at the lower bound, stop or hold the schedule at the declared endpoint; indefinitely extending the cosine formula is not a permanently decreasing rule.^[ref]