 Coupled L2 regularization

Coupled L2 regularization adds a quadratic parameter penalty to the objective function and sends its gradient through the same optimizer update as the data gradient. With coefficient $\lambda \ge 0$, defineSince the gradient of the squared Euclidean norm divided by two is $\theta$, a supplied data gradient $g=\nabla f(\theta )$ becomesThe factor $1/2$ fixes the convention: a penalty written as $\lambda \Vert \theta {\Vert }_2^2$ instead contributes $2\lambda \theta$.

For plain gradient descent, this substitution givesIt is therefore equivalent to decoupled weight decay with the matched coefficient in this particular update. The equivalence need not survive coordinate-dependent processing of the gradient.^[ref]

For an original fixed-scaling example, let $\theta =(1,1{)}^T$, $g=0$, $\eta =0.1$, $\lambda =0.2$ and diagonal matrix $D=\mathrm{diag}(1,0.1)$. Applying $D$ to the whole coupled gradient givesThe same quadratic penalty produces unequal coordinate shrinkage after this scaling. Uniform decoupled weight decay would instead give $(0.98,0.98{)}^T$. The example isolates a fixed scaling operation, rather than claiming to simulate a complete adaptive gradient method.

Coupling can also change the optimizer state. For example, a momentum buffer formed from $\tilde{g}$ stores contributions from $\lambda \theta$ along with the data gradient. If an adaptive accumulator uses squared gradient entries, it receives $(g_i+\lambda {\theta }_i{)}^2$, including a cross term $2\lambda g_i{\theta }_i$. Subsequent updates can therefore differ even when the next data gradient is the same. State both the penalty convention and the optimizer's gradient-processing rule when describing an L2 setting.