Adam 2026-11-05
Adam, named for adaptive moment estimation, is an adaptive gradient method that scales a running gradient estimate by a running squared-gradient estimate. Diederik P. Kingma and Jimmy Lei Ba presented it as a first-order method for stochastic optimization. Its state makes the next optimizer update depend on gradient history as well as the current parameters.[ref]
Starting from , the basic minimization rule is
Here is the supplied gradient, is the learning rate, , and . The Hadamard product, square root and division act coordinatewise. Adam bias correction supplies the two normalization factors. The accumulator concerns a raw second moment, rather than a centered variance.[ref]
For an original scalar calculation, choose , , and , and supply , . The first step has
The second has
Equal and opposite supplied gradients have not undone one another: the running histories give the second step a different magnitude. Plain gradient descent with the same constant learning rate would return to zero for this sequence.
The calculation specifies the recurrence, rather than a convergence guarantee for an unspecified objective function. An optimization checkpoint needs these histories and the step count to reproduce subsequent steps. AdamW adds decoupled weight decay; feeding a penalty gradient into Adam instead implements coupled L2 regularization and changes the histories.
AdamW 2026-11-05
AdamW combines the adaptive update of Adam with decoupled weight decay. It maintains exponentially weighted histories of the supplied data gradient and its componentwise square, while applying parameter shrinkage separately. Ilya Loshchilov and Frank Hutter presented the method in their work on decoupling decay from gradient-based regularization.[ref]
For minimization without optional algorithm variants, initialize . In the convention documented by PyTorch, step uses
The product is a Hadamard product; square roots and division are also coordinatewise. Assume , and the stated learning rate . The decay coefficient is . This explicitly fixes epsilon placement and the decay convention.[ref]
The denominators and perform Adam bias correction for zero-initialized histories. For a constant supplied scalar gradient , the corresponding geometric sums give and . Their corrected values are therefore and . The squared-gradient quantity concerns a raw second moment, rather than a centered variance: a constant gradient of 2 has square 4 even though its variance is zero.
For an original first-step example, choose , , , , , and . Then
and
The constants are chosen for transparent arithmetic. Feeding into the same adaptive formulas instead, with no separate shrinkage, would give approximately . That coupled L2 regularization also changes the stored histories. Resuming AdamW therefore requires the relevant optimizer state and step count as well as the parameter values and settings.
Hadamard power Created 2026-10-06 Updated 2026-11-05
For integer , the entrywise power has entries and equals the repeated Hadamard product. The zero power is the all-ones matrix, even at zero entries, because it represents the constant function one. Every nonnegative integer Hadamard power of a positive semidefinite matrix is positive semidefinite by the Schur product theorem. It is not an ordinary matrix power.
The positive constant in layer normalization regularizes division by the coordinate standard deviation. For coordinates, write , and . The normalized core and the full output are
Here is the Hadamard product, and are fixed when studying the dependence on . This placement of , the divisor in the coordinate variance, and the elementwise layer-normalization gain and layer-normalization bias define the stabilized convention used here.[ref] If all coordinates of are equal, then and : positive makes the normalized core well defined and equal to 0, so the full output is .
Positive also changes the normalization away from constant inputs. The arithmetic mean of the normalized core is 0, but its normalized coordinate variance in layer normalization is
Thus it approaches 1 when is large, equals when , and approaches 0 when approaches 0. This formula describes the normalized core; learned, unequal coordinate gains or biases can change the coordinate arithmetic mean and variance of the full output. For , changing from to changes the core from to .
Although layer normalization is invariant under a common input shift, a fixed positive prevents exact positive scaling invariance on a nonconstant input. For ,
For and fixed , this equals only when . With , the normalized core is exactly invariant for every wherever . The unstabilized scaling argument in Layer Normalization by Jimmy Lei Ba and others therefore needs this qualification when used for the stabilized formula.[ref]
The same constant controls layer-normalization sensitivity. With the centering matrix , we have , and . Taking a derivative gives the layer-normalization Jacobian:
The common-offset vector is an eigenvector with eigenvalue . For , the radial direction has eigenvalue ; a vector orthogonal to both and has eigenvalue . At a constant input, the Jacobian matrix is . These formulas give the operator norm bound for the Euclidean norm. Integrating this derivative along a line segment proves Lipschitz continuity of the normalized core with constant . The bound for the full output is multiplied by , the operator norm of the diagonal matrix of its layer-normalization gain. For , the core bound is attained in centered directions at a constant input: smaller permits larger local amplification near zero coordinate variance.
For the input family , the third coordinate of the normalized core is , with derivative . The dashed lines mark ; the forward response approaches these values for large positive or negative .
Figure 1.
Original layer-normalization response and sensitivity for ; unit gains and zero biases.
Details.
Layer-normalization gain 2026-11-05
The layer-normalization gain is the learned coordinatewise multiplier applied to the normalized core. With layer-normalization bias , the output is , where is the Hadamard product. The multiplier can differ between coordinates and need not be positive.[ref] It permits the model to rescale the features after normalization.
For an original example, choose and layer-normalization epsilon . The coordinate mean is zero, the coordinate variance is , and the normalized core is . Taking and zero bias gives
The output coordinate arithmetic mean is and its coordinate variance is . Thus the core's centering and variance properties do not automatically survive unequal gains. A negative gain also reverses the sign of its coordinate in this example.
If is the incoming loss gradient, then . For above, the gain gradient is . This concerns a derivative with respect to the learned parameter; the layer-normalization Jacobian instead differentiates with respect to the input with parameters fixed.
Setting one to zero makes that output coordinate equal to its bias, but does not remove input coordinate from the shared normalization statistics. It can still influence other output coordinates through those statistics. The input derivative is obtained by multiplying the core Jacobian on the left by the diagonal matrix of gains. Layer-normalization epsilon contains the general derivative and bound.
Schur product theorem Created 2026-09-24 Updated 2026-11-05
The Hadamard product of two complex positive semidefinite matrices of the same size is a positive semidefinite matrix. Write their spectral decompositions as sums and , absorbing each nonnegative eigenvalue into its vector. Then
where is the componentwise product. Every term is a positive semidefinite matrix of matrix rank at most one, so their sum is a positive semidefinite matrix. This proof includes the real case by replacing the conjugate transpose with the transpose. Iterating gives positivity of every nonnegative integer Hadamard power; the zeroth power is the all-ones matrix.