 Adam

Adam, named for adaptive moment estimation, is an adaptive gradient method that scales a running gradient estimate by a running squared-gradient estimate. Diederik P. Kingma and Jimmy Lei Ba presented it as a first-order method for stochastic optimization. Its state makes the next optimizer update depend on gradient history as well as the current parameters.^[ref]

Starting from $m_0=v_0=0$, the basic minimization rule isHere $g_t$ is the supplied gradient, ${\eta }_t>0$ is the learning rate, $0\le {\beta }_1,{\beta }_2<1$, and $ϵ>0$. The Hadamard product, square root and division act coordinatewise. Adam bias correction supplies the two normalization factors. The accumulator $v_t$ concerns a raw second moment, rather than a centered variance.^[ref]

For an original scalar calculation, choose ${\theta }_0=0$, ${\beta }_1={\beta }_2=1/2$, ${\eta }_t=0.1$ and $ϵ=0.1$, and supply $g_1=2$, $g_2=-2$. The first step hasThe second hasEqual and opposite supplied gradients have not undone one another: the running histories give the second step a different magnitude. Plain gradient descent with the same constant learning rate would return to zero for this sequence.

The calculation specifies the recurrence, rather than a convergence guarantee for an unspecified objective function. An optimization checkpoint needs these histories and the step count to reproduce subsequent steps. AdamW adds decoupled weight decay; feeding a penalty gradient into Adam instead implements coupled L2 regularization and changes the histories.