= Weight decay
{title2=$\lambda$}
<Weight decay> supplies a parameter-shrinkage contribution to an <optimizer update>, with a coefficient commonly denoted $\lambda$. Two implementations must be distinguished. <Coupled L2 regularization> adds the penalty $\lambda\lVert\theta\rVert^2/2$ to the <objective function>, replacing its data gradient $g$ by $g+\lambda\theta$. <Decoupled weight decay> applies parameter shrinkage separately from the optimizer's processing of the data gradient. These operations are equivalent for plain <gradient descent> with matched coefficients, but not in general for an <adaptive gradient method>.<Decoupled Weight Decay Regularization by Ilya Loshchilov and Frank Hutter>[Version3, section2, propositions1–2 and AppendixA]{ref}
Use the <PyTorch> convention here: a decoupled update shrinks parameters by $1-\eta\lambda$, where $\eta$ is the <learning rate>. A source writing $1-\lambda$ instead has absorbed the step-size factor into its decay coefficient. For an unmodified gradient step,
$$
\theta-\eta(g+\lambda\theta)=(1-\eta\lambda)\theta-\eta g.
$$
In <AdamW>, the decay contribution does not enter the stored first- or second-moment estimates of the gradient.<AdamW in the PyTorch documentation>[PyTorch2.15, algorithm, separate parameter-decay line and moment updates]{ref}
An original fixed-scaling example exposes the difference. Let $\theta=(2,2)^T$, $g=(1,1)^T$, $\eta=0.1$, $\lambda=0.2$ and <diagonal matrix> $A=\operatorname{diag}(1,10)$. Applying the same scaling to the entire coupled gradient gives
$$
\theta_{\rm coupled}=\theta-\eta A(g+\lambda\theta)=(1.86,0.60)^T.
$$
Separating uniform parameter shrinkage gives
$$
\theta_{\rm decoupled}=(1-\eta\lambda)\theta-\eta Ag=(1.86,0.96)^T.
$$
The second coordinate receives different shrinkage. This is an illustrative fixed matrix, not a full AdamW simulation. With history-dependent scaling, coupling the penalty into the gradient can also alter the <optimizer state>. State the update equation and coefficient convention when reporting a decay setting; the label alone does not specify either.
Back to article page
