= Coupled L2 regularization
{title2=$g\mapsto g+\lambda\theta$}
\b[Coupled L2 regularization] adds a quadratic parameter penalty to the <objective function> and sends its <gradient> through the same <optimizer update> as the data gradient. With coefficient $\lambda\geq0$, define
$$
F(\theta)=f(\theta)+\frac{\lambda}{2}\lVert\theta\rVert_2^2.
$$
Since the <gradient> of the <squared Euclidean norm> divided by two is $\theta$, a supplied data gradient $g=\nabla f(\theta)$ becomes
$$
\widetilde g=g+\lambda\theta.
$$
The factor $1/2$ fixes the convention: a penalty written as $\lambda\lVert\theta\rVert_2^2$ instead contributes $2\lambda\theta$.
For plain <gradient descent>, this substitution gives
$$
\theta^+=\theta-\eta(g+\lambda\theta)
=(1-\eta\lambda)\theta-\eta g.
$$
It is therefore equivalent to <decoupled weight decay> with the matched coefficient in this particular update. The equivalence need not survive coordinate-dependent processing of the gradient.<Decoupled Weight Decay Regularization by Ilya Loshchilov and Frank Hutter>[Version3, section2, propositions1–2 and AppendixA]{ref}
For an original fixed-scaling example, let $\theta=(1,1)^T$, $g=0$, $\eta=0.1$, $\lambda=0.2$ and <diagonal matrix> $D=\operatorname{diag}(1,0.1)$. Applying $D$ to the whole coupled gradient gives
$$
\theta^+=\theta-\eta D(\lambda\theta)
=(0.98,0.998)^T.
$$
The same quadratic penalty produces unequal coordinate shrinkage after this scaling. Uniform <decoupled weight decay> would instead give $(0.98,0.98)^T$. The example isolates a fixed scaling operation, rather than claiming to simulate a complete <adaptive gradient method>.
Coupling can also change the <optimizer state>. For example, a <momentum buffer> formed from $\widetilde g$ stores contributions from $\lambda\theta$ along with the data gradient. If an adaptive accumulator uses squared gradient entries, it receives $(g_i+\lambda\theta_i)^2$, including a cross term $2\lambda g_i\theta_i$. Subsequent updates can therefore differ even when the next data gradient is the same. State both the penalty convention and the optimizer's gradient-processing rule when describing an L2 setting.
Back to article page
