{c}

\b[AdamW] combines the adaptive update of <Adam> with <decoupled weight decay>. It maintains exponentially weighted histories of the supplied data <gradient> and its componentwise square, while applying parameter shrinkage separately. <Ilya Loshchilov> and <Frank Hutter> presented the method in their work on decoupling decay from gradient-based regularization.<Decoupled Weight Decay Regularization by Ilya Loshchilov and Frank Hutter>[Version3, Algorithm2, AdamW]{ref}

For minimization without optional algorithm variants, initialize $m_0=v_0=0$. In the convention documented by <PyTorch>, step $t\geq1$ uses
$$
\begin{aligned}
g_t&=\nabla f_t(\theta_{t-1}),\\
m_t&=\beta_1m_{t-1}+(1-\beta_1)g_t,\\
v_t&=\beta_2v_{t-1}+(1-\beta_2)(g_t\odot g_t),\\
\widehat m_t&=\frac{m_t}{1-\beta_1^t},\qquad
\widehat v_t=\frac{v_t}{1-\beta_2^t},\\
\theta_t&=(1-\eta_t\lambda)\theta_{t-1}
-\eta_t\frac{\widehat m_t}{\sqrt{\widehat v_t}+\epsilon}.
\end{aligned}
$$
The product $\odot$ is a <Hadamard product>; square roots and division are also coordinatewise. Assume $0\leq\beta_1,\beta_2<1$, $\epsilon>0$ and the stated <learning rate> $\eta_t$. The decay coefficient is $\lambda$. This explicitly fixes epsilon placement and the decay convention.<AdamW in the PyTorch documentation>[PyTorch2.15, algorithm with amsgrad and maximize disabled]{ref}

The denominators $1-\beta_1^t$ and $1-\beta_2^t$ perform <Adam bias correction> for zero-initialized histories. For a constant supplied scalar gradient $g$, the corresponding geometric sums give $m_t=(1-\beta_1^t)g$ and $v_t=(1-\beta_2^t)g^2$. Their corrected values are therefore $g$ and $g^2$. The squared-gradient quantity concerns a raw <second moment>, rather than a centered <variance>: a constant gradient of 2 has square 4 even though its variance is zero.

For an original first-step example, choose $f(\theta)=\theta^2$, $\theta_0=1$, $\eta=0.1$, $\lambda=0.2$, $\beta_1=0.9$, $\beta_2=0.99$ and $\epsilon=0.1$. Then
$$
g_1=2,\quad m_1=0.2,\quad v_1=0.04,\quad
\widehat m_1=2,\quad\widehat v_1=4,
$$
and
$$
\theta_1=0.98-0.1\frac{2}{2+0.1}
=\frac{929}{1050}\approx0.884762.
$$
The constants are chosen for transparent arithmetic. Feeding $g_1+\lambda\theta_0=2.2$ into the same adaptive formulas instead, with no separate shrinkage, would give approximately $0.904348$. That <coupled L2 regularization> also changes the stored histories. Resuming AdamW therefore requires the relevant <optimizer state> and step count as well as the parameter values and settings.


 AdamW (source code)