Adam

Adam, named for adaptive moment estimation, is an adaptive gradient method that scales a running gradient estimate by a running squared-gradient estimate. Diederik P. Kingma and Jimmy Lei Ba presented it as a first-order method for stochastic optimization. Its state makes the next optimizer update depend on gradient history as well as the current parameters.^[ref]

Starting from $m_0=v_0=0$, the basic minimization rule isHere $g_t$ is the supplied gradient, ${\eta }_t>0$ is the learning rate, $0\le {\beta }_1,{\beta }_2<1$, and $ϵ>0$. The Hadamard product, square root and division act coordinatewise. Adam bias correction supplies the two normalization factors. The accumulator $v_t$ concerns a raw second moment, rather than a centered variance.^[ref]

For an original scalar calculation, choose ${\theta }_0=0$, ${\beta }_1={\beta }_2=1/2$, ${\eta }_t=0.1$ and $ϵ=0.1$, and supply $g_1=2$, $g_2=-2$. The first step hasThe second hasEqual and opposite supplied gradients have not undone one another: the running histories give the second step a different magnitude. Plain gradient descent with the same constant learning rate would return to zero for this sequence.

The calculation specifies the recurrence, rather than a convergence guarantee for an unspecified objective function. An optimization checkpoint needs these histories and the step count to reproduce subsequent steps. AdamW adds decoupled weight decay; feeding a penalty gradient into Adam instead implements coupled L2 regularization and changes the histories.

Adam bias correction compensates for the initial zero values in Adam's exponentially weighted gradient histories. The basic algorithm usesThese factors normalize the coefficient sums of the two running histories.^[ref]

For a scalar coordinate, suppose $m_0=0$ andExpanding the recurrence relation givesIf every $g_s$ has the same expectation $\mu$, linearity of expectation and the geometric series yieldThus ${\hat{m}}_t$ is an unbiased estimator of $\mu$ under these assumptions. Correlation between different times is allowed by this argument. Applying the same calculation to squared gradients with constant raw second moment $\nu =\mathbb{E}[g_s^2]$ gives $\mathbb{E}[v_t]=(1-{\beta }_2^t)\nu$ and $\mathbb{E}[{\hat{v}}_t]=\nu$.

For a constant supplied gradient of 2, choose ${\beta }_1=0.9$ and ${\beta }_2=0.99$. The first uncorrected histories are $m_1=0.2$ and $v_1=0.04$, while their corrected values are 2 and 4. The normalization removes the missing coefficient mass caused by initialization.

It does not make a changing history equal to the latest gradient. For an original deterministic example, let ${\beta }_1=1/2$, $g_1=1$ and $g_2=3$. Then $m_2=7/4$, sowhich differs from $g_2=3$. More generally, changing moment expectations produce a weighted average of past expectations. State the moment assumptions before interpreting “bias-corrected” as unbiased for a present distribution. The nonlinear parameter update built from the corrected histories is a separate object from either moment estimator.