= Decoupled weight decay
{title2=$\theta\mapsto(1-\eta\lambda)\theta$}
\b[Decoupled weight decay] applies a parameter-shrinkage contribution separately from the optimizer's processing of the data <gradient>. Write $u_t$ for the processed data-gradient direction, including any algorithm-specific <optimizer state>. Using the coefficient convention documented for <PyTorch> <AdamW>,
$$
\theta_t=(1-\eta_t\lambda)\theta_{t-1}-\eta_t u_t.
$$
The shrinkage term uses the pre-update parameter and is not inserted into the gradient moments.<AdamW in the PyTorch documentation>[PyTorch2.15, algorithm: separate parameter-decay line before the adaptive update]{ref}
This differs from <coupled L2 regularization>, which processes $g_t+\lambda\theta_{t-1}$ as a single gradient input. <Decoupled weight decay> was introduced for adaptive optimization in the AdamW work; the original paper also writes decay coefficients with the learning-rate factor absorbed differently, so its coefficient must be translated before comparing equations.<Decoupled Weight Decay Regularization by Ilya Loshchilov and Frank Hutter>[Version3, section2 and Algorithm2]{ref}
An original limiting case isolates the shrinkage. Suppose the processed direction is zero at every step and $\eta,\lambda$ are constant. Repeated substitution gives
$$
\theta_T=(1-\eta\lambda)^T\theta_0.
$$
With $\eta=0.1$, $\lambda=0.2$ and $\theta_0=1$, one update gives 0.98 and 50 updates give approximately 0.36417. For varying <learning rates>, the corresponding factor is
$$
\theta_T=\left(\prod_{t=1}^{T}(1-\eta_t\lambda)\right)\theta_0.
$$
Thus the same numerical decay coefficient can have a different accumulated effect under a different <learning rate schedule>. In this zero-direction model, $0<\eta\lambda<1$ gives sign-preserving shrinkage; other factors have different behaviour.
The update order is also part of the definition. For $\theta=1$, $u=2$, $\eta=0.1$ and $\lambda=0.2$, the stated formula gives $0.98-0.20=0.78$. Applying shrinkage after first taking the gradient step would instead give $(1-0.20)0.98=0.784$. Both operations may be described informally as decay plus a step, but they are different recurrences. An <optimization checkpoint> must retain the relevant state and settings to reproduce the chosen one.
Back to article page
