= Learning rate
{title2=$\eta$}
A <learning rate> controls the size of an iterative learning update. In <gradient descent>, it scales a direction obtained from the <gradient> of a loss:
$$
\theta_{t+1}=\theta_t-\eta_t\nabla L(\theta_t).
$$
A <stochastic gradient descent> update uses an estimated gradient in this expression. The documented SGD algorithm makes the learning rate an explicit multiplier; momentum and other modifications add separate behavior.<SGD in the PyTorch documentation>[Algorithm pseudocode and lr parameter]{ref}
A scalar example shows why size matters. For $L(\theta)=\theta^2/2$, the gradient is $\theta$, so a constant learning rate gives
$$
\theta_{t+1}=(1-\eta)\theta_t,\qquad
\theta_t=(1-\eta)^t\theta_0.
$$
For nonzero initial value, convergence to zero occurs exactly when $|1-\eta|<1$, or $0<\eta<2$. With $\theta_0=4$ and $\eta=1/2$, the values are $4,2,1,1/2,\ldots$. With $\eta=5/2$, they begin $4,-6,9,-13.5,\ldots$: increasing the step has made the iteration diverge rather than merely learn faster.
This threshold belongs to the stated quadratic. Changing curvature changes the allowed scale: for $L(\theta)=a\theta^2/2$ with $a>0$, the same proof gives $0<\eta<2/a$. A numerical learning rate therefore has meaning only together with the objective and update rule.
In <reinforcement learning>, an incremental value estimate can instead use the update $Q\leftarrow Q+\alpha(r-Q)$. For an illustrative $Q=2$, observed target $r=4$ and $\alpha=0.1$, the new estimate is $2.2$. Here the learning rate weights new evidence against the previous estimate, showing the same step-size idea without requiring a gradient calculation.
A <learning rate schedule> can vary the rate with iteration. Report the schedule, objective scaling and other update settings when comparing results; the rate alone does not determine stability or final accuracy.
Back to article page
