Squared-error backpropagation Created 2026-10-14 Updated 2026-10-25
For squared loss , let and . The chain rule gives and . A gradient descent step of size uses and , with every derivative evaluated at the same pre-update weights.
Symmetric preconditioning as energy geometry 2026-10-18
For a symmetric positive-definite matrix , solving is equivalent to minimizing the quadratic form with linear termWriting the error vector and expanding gives the exact energy norm identityThe cross term vanishes because . With and , the same quantity is . Symmetric positive-definite preconditioning changes coordinates without changing the energy assigned to a corresponding error vector.
In an orthonormal eigenbasis of , the level set has semiaxis lengths . Its longest-to-shortest axis ratio is . A large spectral condition number of a positive-definite matrix therefore appears geometrically as a thin ellipsoid. Good scaling makes the transformed ellipsoid rounder, allowing a simple search direction to improve all coordinates together.
The negative gradient of is the linear-system residual . Relative to the norm , a steepest direction is : the Cauchy-Schwarz inequality applied to shows that the greatest decrease per unit -length occurs along this direction. Equivalently, ordinary gradient descent in becomes preconditioned steepest descent in .[ref]
Warm restart (optimization) 2026-11-05
A warm restart continues an optimization process from an already obtained solution while restarting specified algorithm controls or history. Its meaning depends on which fields are retained. In SGDR, the current parameters are retained while the learning rate schedule starts another cycle; the original paper explicitly distinguishes this from starting the optimization from scratch.[ref]
SGDR: Stochastic Gradient Descent with Warm Restarts by Ilya Loshchilov and Frank Hutter. Version5, section3, printed p4, retained current solution and restarted learning rate
Write a complete algorithm state as , where holds parameters, is optimizer state and holds controls such as the schedule phase. A restart map can retain while replacing , or can also replace selected parts of . Saying only “restart” does not specify the next optimizer update.
For an original scalar example, take and the momentum buffer recurrenceAt current , stored , and restarted learning rate , retaining the buffer givesRetaining the same parameters but resetting the buffer to zero gives and . Both runs start the next update at ; they differ in the history carried through the restart.
A schedule-only restart can also change the next step without changing stored parameters immediately. In a plain gradient descent update from , increasing the learning rate from 0.01 to 0.1 changes the next value from 1.98 to 1.8. The change takes effect through the subsequent gradient step.
An optimization checkpoint instead records enough information to reproduce a specified continuation. A deliberate warm restart may intentionally alter some of that information. Record the retained parameter state, the new schedule phase and the policy for buffers or counters before comparing trajectories or claiming optimizer restart equivalence.
Weight decay 2026-11-05
Weight decay supplies a parameter-shrinkage contribution to an optimizer update, with a coefficient commonly denoted . Two implementations must be distinguished. Coupled L2 regularization adds the penalty to the objective function, replacing its data gradient by . Decoupled weight decay applies parameter shrinkage separately from the optimizer's processing of the data gradient. These operations are equivalent for plain gradient descent with matched coefficients, but not in general for an adaptive gradient method.[ref]
Decoupled Weight Decay Regularization by Ilya Loshchilov and Frank Hutter. Version3, section2, propositions1–2 and AppendixA
Use the PyTorch convention here: a decoupled update shrinks parameters by , where is the learning rate. A source writing instead has absorbed the step-size factor into its decay coefficient. For an unmodified gradient step,In AdamW, the decay contribution does not enter the stored first- or second-moment estimates of the gradient.[ref]
AdamW in the PyTorch documentation. PyTorch2.15, algorithm, separate parameter-decay line and moment updates
An original fixed-scaling example exposes the difference. Let , , , and diagonal matrix . Applying the same scaling to the entire coupled gradient givesSeparating uniform parameter shrinkage givesThe second coordinate receives different shrinkage. This is an illustrative fixed matrix, not a full AdamW simulation. With history-dependent scaling, coupling the penalty into the gradient can also alter the optimizer state. State the update equation and coefficient convention when reporting a decay setting; the label alone does not specify either.
Yoram Singer 2026-11-05
Yoram Singer coauthored the 2011 AdaGrad paper with John Duchi and Elad Hazan. Their adaptive gradient methods use past subgradients to change the geometry of later optimizer updates.[ref]
An unconstrained diagonal specialization uses a diagonal AdaGrad accumulatorwith learning rate and stabilization outside the square root. The accumulator is part of the optimizer state.[ref]
For an original supplied-gradient example, let the current accumulator be , the current gradient be , and choose , . These accumulator values include the current gradient. The next displacement isThe identical current gradient coordinates produce different movements because their histories differ. Plain gradient descent with the same learning rate would move by . This calculation exposes the mechanism without asserting that either update improves every possible objective function.
The example also explains why saving only the parameter vector is insufficient to reproduce a later adaptive step. Two runs at the same , receiving the same , can move differently if their stored accumulators differ.
Yoram Singer, portrait linked from his personal research homepage
. [ref]Yoram Singer personal research homepage. Named homepage and linked portrait
