The learning rate is a hyperparameter used in optimization algorithms, particularly in the context of machine learning and neural networks. It controls how much to change the model weights in response to the error or loss calculated during training. In more specific terms, the learning rate determines the size of the steps taken towards a minimum of the loss function during the training process.
Articles by others on the same topic
A learning rate controls the size of an iterative learning update. In gradient descent, it scales a direction obtained from the gradient of a loss:A stochastic gradient descent update uses an estimated gradient in this expression. The documented SGD algorithm makes the learning rate an explicit multiplier; momentum and other modifications add separate behavior.[ref]
A scalar example shows why size matters. For , the gradient is , so a constant learning rate givesFor nonzero initial value, convergence to zero occurs exactly when , or . With and , the values are . With , they begin : increasing the step has made the iteration diverge rather than merely learn faster.
This threshold belongs to the stated quadratic. Changing curvature changes the allowed scale: for with , the same proof gives . A numerical learning rate therefore has meaning only together with the objective and update rule.
In reinforcement learning, an incremental value estimate can instead use the update . For an illustrative , observed target and , the new estimate is . Here the learning rate weights new evidence against the previous estimate, showing the same step-size idea without requiring a gradient calculation.
A learning rate schedule can vary the rate with iteration. Report the schedule, objective scaling and other update settings when comparing results; the rate alone does not determine stability or final accuracy.
