= Learning rate schedule
{title2=$\eta_t$}
A <learning rate schedule> specifies how the <learning rate> changes during an iterative learning procedure. Its rule can depend on the update count, a <training epoch> count or another declared event. The rule and the clock must both be stated: changing a rate every parameter update is different from changing it after a complete pass through the training data.
A simple step-decay example is
$$
\eta(t)=0.1(0.1)^{\lfloor t/3\rfloor},
$$
where $t$ counts completed schedule advances from zero. The values at $t=0,1,2$ are $0.1$, those at $t=3,4,5$ are $0.01$, and the value at $t=6$ is $0.001$. The StepLR implementation documents periodic multiplication by a specified factor; its example advances the scheduler after the training and validation operations.<StepLR in the PyTorch documentation>[Class description, step_size/gamma parameters and example loop]{ref}
A different schedule admits an exact convergence calculation. For <gradient descent> on $L(\theta)=\theta^2/2$, take $\eta_t=1/(t+2)$. With no other update terms,
$$
\theta_{t+1}=\theta_t-\eta_t\theta_t
=\frac{t+1}{t+2}\theta_t.
$$
Multiplying the successive ratios gives a telescoping product:
$$
\theta_t=\theta_0\prod_{j=0}^{t-1}\frac{j+1}{j+2}
=\frac{\theta_0}{t+1}.
$$
Starting at $\theta_0=4$ gives $4,2,4/3,1,\ldots$. This proves convergence for the stated deterministic quadratic, rather than for every objective or noisy learning procedure.
The underlying <stochastic gradient descent> update also depends on its gradient estimate and any additional terms, not just the scheduled scalar.<SGD in the PyTorch documentation>[Algorithm pseudocode and lr parameter]{ref} Report the initial index, update order and any external changes to the rate. The StepLR documentation explicitly permits other changes alongside its own, so the isolated step formula above assumes that no competing rule intervenes.<StepLR in the PyTorch documentation>[Class description, external learning-rate changes]{ref}
Back to article page
