Adam (source code)
= Adam
{c}
\b[Adam], named for adaptive moment estimation, is an <adaptive gradient method> that scales a running gradient estimate by a running squared-gradient estimate. <Diederik P. Kingma> and <Jimmy Lei Ba> presented it as a first-order method for stochastic optimization. Its state makes the next <optimizer update> depend on gradient history as well as the current parameters.<Adam: A Method for Stochastic Optimization by Diederik P. Kingma and Jimmy Lei Ba>[Version9, printed pp1–2, Introduction and Algorithm1]{ref}
Starting from $m_0=v_0=0$, the basic minimization rule is
$$
\begin{aligned}
m_t&=\beta_1m_{t-1}+(1-\beta_1)g_t,\\
v_t&=\beta_2v_{t-1}+(1-\beta_2)(g_t\odot g_t),\\
\widehat m_t&=m_t/(1-\beta_1^t),\qquad
\widehat v_t=v_t/(1-\beta_2^t),\\
\theta_t&=\theta_{t-1}-\eta_t
\frac{\widehat m_t}{\sqrt{\widehat v_t}+\epsilon}.
\end{aligned}
$$
Here $g_t$ is the supplied <gradient>, $\eta_t>0$ is the <learning rate>, $0\leq\beta_1,\beta_2<1$, and $\epsilon>0$. The <Hadamard product>, square root and division act coordinatewise. <Adam bias correction> supplies the two normalization factors. The accumulator $v_t$ concerns a raw <second moment>, rather than a centered <variance>.<Adam: A Method for Stochastic Optimization by Diederik P. Kingma and Jimmy Lei Ba>[Algorithm1, printed p2, epsilon outside the square root]{ref}
For an original scalar calculation, choose $\theta_0=0$, $\beta_1=\beta_2=1/2$, $\eta_t=0.1$ and $\epsilon=0.1$, and supply $g_1=2$, $g_2=-2$. The first step has
$$
m_1=1,\quad v_1=2,\quad
\widehat m_1=2,\quad\widehat v_1=4,\quad
\theta_1=-\frac{2}{21}.
$$
The second has
$$
m_2=-\frac12,\quad v_2=3,\quad
\widehat m_2=-\frac23,\quad\widehat v_2=4,
\qquad
\theta_2=-\frac{2}{21}+\frac{2}{63}=-\frac4{63}.
$$
Equal and opposite supplied gradients have not undone one another: the running histories give the second step a different magnitude. Plain <gradient descent> with the same constant learning rate would return to zero for this sequence.
The calculation specifies the recurrence, rather than a convergence guarantee for an unspecified <objective function>. An <optimization checkpoint> needs these histories and the step count to reproduce subsequent steps. <AdamW> adds <decoupled weight decay>; feeding a penalty gradient into Adam instead implements <coupled L2 regularization> and changes the histories.
Back to article page
