Adam 2026-11-05
Adam, named for adaptive moment estimation, is an adaptive gradient method that scales a running gradient estimate by a running squared-gradient estimate. Diederik P. Kingma and Jimmy Lei Ba presented it as a first-order method for stochastic optimization. Its state makes the next optimizer update depend on gradient history as well as the current parameters.[ref]
Starting from , the basic minimization rule is
Here is the supplied gradient, is the learning rate, , and . The Hadamard product, square root and division act coordinatewise. Adam bias correction supplies the two normalization factors. The accumulator concerns a raw second moment, rather than a centered variance.[ref]
For an original scalar calculation, choose , , and , and supply , . The first step has
The second has
Equal and opposite supplied gradients have not undone one another: the running histories give the second step a different magnitude. Plain gradient descent with the same constant learning rate would return to zero for this sequence.
The calculation specifies the recurrence, rather than a convergence guarantee for an unspecified objective function. An optimization checkpoint needs these histories and the step count to reproduce subsequent steps. AdamW adds decoupled weight decay; feeding a penalty gradient into Adam instead implements coupled L2 regularization and changes the histories.
Adaptive gradient method 2026-11-05
An adaptive gradient method uses information from supplied gradients to adjust subsequent optimizer updates, often with a different scale for each coordinate. A useful diagonal form is
where is the current gradient, is a scalar learning rate and is a positive diagonal matrix computed from gradient history. A learning rate schedule can still control independently of the coordinate adaptation. The broader family includes AdaGrad, Adam and AdamW.
The 2011 AdaGrad paper by John Duchi, Elad Hazan and Yoram Singer formalized adaptive geometry from accumulated gradient information, including diagonal and full-matrix constructions.[ref] An unconstrained diagonal specialization maintains
Here specifies additive stabilization outside the square root. The accumulator is part of the optimizer state, rather than determined by the current parameters alone.
For an original example, start at and supply , with and . Both accumulated entries are positive, so the denominators are defined. The first step gives
Plain gradient descent with the same scalar learning rate would instead give . If the next supplied gradient is again , then and each coordinate's additional movement is . These are explicit supplied-gradient calculations, rather than a convergence claim.
In this cumulative version, an accumulated squared-gradient entry never decreases, so its effective coordinate learning rate does not increase. Other adaptive gradient methods retain decaying averages instead, changing how strongly old gradients influence later updates. Adaptation therefore specifies both the state recurrence and how that state transforms the gradient; the word “adaptive” alone does not determine either.
Coupled L2 regularization 2026-11-05
Coupled L2 regularization adds a quadratic parameter penalty to the objective function and sends its gradient through the same optimizer update as the data gradient. With coefficient , define
Since the gradient of the squared Euclidean norm divided by two is , a supplied data gradient becomes
The factor fixes the convention: a penalty written as instead contributes .
For plain gradient descent, this substitution gives
It is therefore equivalent to decoupled weight decay with the matched coefficient in this particular update. The equivalence need not survive coordinate-dependent processing of the gradient.[ref]
For an original fixed-scaling example, let , , , and diagonal matrix . Applying to the whole coupled gradient gives
The same quadratic penalty produces unequal coordinate shrinkage after this scaling. Uniform decoupled weight decay would instead give . The example isolates a fixed scaling operation, rather than claiming to simulate a complete adaptive gradient method.
Coupling can also change the optimizer state. For example, a momentum buffer formed from stores contributions from along with the data gradient. If an adaptive accumulator uses squared gradient entries, it receives , including a cross term . Subsequent updates can therefore differ even when the next data gradient is the same. State both the penalty convention and the optimizer's gradient-processing rule when describing an L2 setting.
Delta rule 2026-10-25
For a linear output and squared loss , gradient descent gives . Indeed . On the same example, , proving strict error reduction when and .
Derivative Created 2026-09-24 Updated 2026-10-14
The derivative measures the instantaneous rate at which a function changes. For a real function defined near , it exists when the difference quotient has a finite two-sided limit:
The quotient is the slope of the straight line joining two nearby points of the graph. Its limit is the slope of the tangent line. Thus a derivative describes a local rate without choosing a particular finite measurement interval.
For , expanding the numerator gives
At , the increments , and give secant line slopes , and . The limit is , and the tangent line is . The picture makes the limiting process visible.
Figure 1.
Secant lines to x squared approach its tangent line at x = 1
. Details.
The same calculation shows why the derivative is useful for approximation:
This is a local linear approximation. Near , replacing by makes the exact error , smaller than the leading change when is small. A curved graph can therefore be studied locally using a straight line.
If the function records position against time, its derivative is velocity. If it records an objective function, its derivative tests local changes that matter in optimization; in several variables, the gradient collects these directional rates and guides gradient descent. The interpretation comes from the function and its units, rather than from a new rule for each application.
Existence matters. For the absolute value function , the one-sided derivatives at zero are and , so there is no single tangent line described by a finite derivative there. Nor does a zero derivative alone identify a maximum or minimum: has but keeps increasing through zero.
Descent lemma Created 2026-10-06 Updated 2026-10-25
Suppose is a differentiable function on an open convex set and has an -Lipschitz gradient there, in the Euclidean norm. For , the descent lemma states
A bounded continuous Hessian matrix gives one sufficient route to this estimate.[ref]
For a short proof, put . The fundamental theorem of calculus along the line segment gives
The Cauchy-Schwarz inequality and the Lipschitz gradient bound make the integrand at most ; its integral is . This proof needs no assumption that is a convex function.
For the gradient descent update , provided , substitution yields
For and , this ensures strict decrease whenever the gradient is nonzero. At it guarantees only nonincrease; the quadratic example has a nonshrinking alternating component at this endpoint. At the guaranteed decrease is . A bound valid only on cannot justify a step leaving .
If a fixed admissible step size is used forever, every iterate stays in , and has a finite lower bound, summing this inequality shows , hence . That conclusion alone gives neither convergence of the iterates nor a global minimum. For example, the smooth polynomial has and : gradient descent started at zero never moves, although . On , , so the Lipschitz gradient hypothesis holds on this domain. Convex functions and strongly convex functions provide additional structure; the exact quadratic convergence result above uses a positive-definite matrix.
Epoch complexity 2026-10-25
The number of optimization epochs needed to meet a stated accuracy criterion. Comparisons must specify the work counted by an optimization epoch, the accuracy measure, and whether a random method is controlled in expectation or with a stated probability. Counting coordinate partial derivatives is useful for comparing randomized block coordinate descent with gradient descent, but need not match elapsed computation time.
Target: gradient-descent.bigb, under Stochastic gradient descent.
Fixed-step gradient descent on a positive-definite quadratic Created 2026-10-18 Updated 2026-10-25
Consider the original two-dimensional example
The Hessian matrix is , a symmetric matrix and positive-definite matrix. Its eigenvectors are the coordinate directions, its eigenvalues are and , and its Euclidean norm condition number is . The unique global minimum is , so the error equals . The linear map , with the identity matrix, gives
For , the two coordinate error multipliers are and . Consequently , , and the second coordinate changes sign at every step while both shrink. The Euclidean norm of the error contracts by each step. The objective gap, which we call the quadratic energy, contracts by its square:
The level sets of are ellipses. Oscillation across their narrow direction is compatible with strictly decreasing energy.
Figure 1.
Original gradient descent iterates on : step size contracts both directions, whereas expands the second
. Details.
For , the coordinate error multipliers become and , giving . The first coordinate still tends to zero, but the second alternates and grows. Direct substitution gives
Thus a larger positive step size can turn a strictly convex function into a divergent iteration. Energy and error are different quantities: coordinate errors carry signed multipliers, while each coordinate's contribution to energy carries the squared multiplier.
Figure 2.
Exact constructed quadratic-energy histories from : step sizes and converge, while diverges
. Details.
For the general quadratic form with linear term
the error obeys the same linear map . The spectral theorem for real symmetric matrices supplies an orthonormal basis of eigenvectors with positive eigenvalues . Writing gives the exact solution and energy identity
Therefore convergence from every initial vector is equivalent to
At , a nonzero error component along a largest-eigenvalue eigenvector flips sign without shrinking. Above that threshold it grows. A particular start can still converge if it has zero components in every unstable direction; the interval is the condition for all starts. If , then and . These bounds follow by summing the squared components in the orthonormal basis.
Gradient Created 2026-09-24 Updated 2026-10-14
In Cartesian coordinates on Euclidean space, the gradient of a scalar-valued differentiable function is the vector of its first partial derivatives. It converts a scalar field, such as temperature or a height, into a vector field describing its local change.
For a unit vector , the directional derivative is . The Cauchy-Schwarz inequality gives . At a point where , the greatest increase per unit distance occurs along , while the opposite direction gives the greatest decrease. These are first-order statements at the current point; they do not predict how the function behaves far along a straight path.
For at , . Moving in the direction gives rate and moving in the direction gives rate , but the steepest direction is , with
The direction has rate zero and is tangent to the level curve through that point. More generally, differentiating along a regular level curve gives , so the gradient is its normal vector.
Figure 1.
A gradient is normal to level curves and determines the directional derivative for every unit direction
. Details.
This is useful when navigating a measured scalar field or applying gradient descent: one collection of partial derivatives predicts every infinitesimal directional change. The derivative assumes a differentiable function, not merely the existence of all directional derivatives. [ref].
Gradient ascent Created 2026-10-14 Updated 2026-10-18
Gradient ascent seeks a maximum by stepping along the gradient of a differentiable objective. It is gradient descent applied to . Taylor expansion gives , so sufficiently small positive steps improve the objective when the gradient is nonzero. Line search, projection onto admissible controls, and noise-aware gradient estimates are common practical modifications; a local stationary result need not be globally optimal.
Gradient descent Created 2026-09-24 Updated 2026-10-18
Gradient descent is an iterative method for minimizing a differentiable function : choose an initial vector and repeatedly move opposite its gradient.[ref]
With a fixed positive step size , the update is
The direction and the step size play different roles: the gradient supplies local information about increase, while determines how far that information is trusted. A varying step size gives instead; exact line search for a positive-definite quadratic chooses it by minimizing along the current direction.
Why is this the steepest direction? For a unit vector in the Euclidean norm, the directional derivative is . The Cauchy-Schwarz inequality bounds it below by , attained by when the gradient is nonzero. This is steepest descent relative to the Euclidean norm; changing the norm can change the steepest direction. A finite step need not decrease , because a directional derivative describes infinitesimal motion.
For the quadratic form with linear term , assuming is a symmetric matrix, the negative gradient is the linear-system residual . Thus gradient descent turns the linear system into repeated residual corrections; a positive-definite matrix makes its solution the unique global minimum. This is useful when applying the linear map is affordable but directly solving the linear system is costly.
The attraction is that a gradient evaluation gives an entire update without solving a new linear system. The difficulty is choosing a step size that progresses along gently curved directions without overshooting steep ones. Fixed-step gradient descent on a positive-definite quadratic makes this tension exact: every eigenvector direction has its own multiplier, determined by its eigenvalue. The resulting condition number dependence explains why a narrow valley can slow even an inexpensive iterative method.
The simplicity of the gradient update and its sensitivity to the condition number of the Hessian matrix are central practical features.[ref]
Gradient descent preserves an eigenline for a cubic product Created 2026-10-25 Updated 2026-10-31
Let be real symmetric, let be a unit eigenvector satisfying , and use scalar step sizes . The gradient of the displayed cubic polynomial function is
On the eigenline, , so a gradient descent update remains on that same line and obeys the scalar recurrence in the title. Induction proves invariance for every iterate starting on the line, including an iterate that reaches zero. This is a statement about the geometry of the updates, not a convergence theorem: when , the restriction is unbounded below in one direction.
Gradient flow Created 2026-09-24 Updated 2026-11-05
For a smooth real function on Euclidean space, its downward gradient flow is the ordinary differential equation . The chain rule gives
Thus is a Lyapunov function, and a nonstationary trajectory cannot return to its starting point. For , the solution is . The explicit Euler method gives , namely gradient descent. The continuous decrease does not guarantee decrease for every discrete step size; even this quadratic example requires for strict contraction.
Ilya Loshchilov 2026-11-05
Ilya Loshchilov is a researcher in optimization and machine learning whose work with Frank Hutter includes SGDR and AdamW. His Freiburg institutional profile identifies the same author and lists the two papers; it is an alumni record, rather than evidence of a current appointment.[ref]
Their 2017 SGDR paper introduced a learning rate schedule that decreases the rate along a cosine curve, then raises it at a warm restart while retaining the current parameters. In one cycle,
Here is elapsed cycle progress and its length. With and , the rates at are . Raising the rate starts another exploration from the learned parameters; it does not reinitialize the model.[ref]
Their 2019 AdamW work separates decoupled weight decay from the gradient-based update. For ordinary gradient descent, adding gives . Coordinate-dependent scaling can act differently on the penalty contribution, so that equivalence does not extend unchanged to an adaptive gradient method.[ref] Writing the equation makes the contribution clearer than comparing optimizer names alone.
Figure 1.
Ilya Loshchilov, portrait linked from his Freiburg Machine Learning Lab alumni profile
.
Landweber iteration Created 2026-09-24 Updated 2026-11-05
For a bounded linear operator between Hilbert spaces, Landweber iteration applies gradient descent to the squared residual:
For nonzero , the standard uniform step range is . Starting at zero gives
The iteration introduces no component in . On data admitting a least-squares solution, it converges to the minimum-norm least-squares solution in this step range.
A singular value has error multiplier . Small singular values converge slowly, so a finite iteration admits well-determined components before aggressively fitting poorly determined ones. Continuing indefinitely on noisy data can amplify precisely those late components. Early stopping of Landweber iteration therefore acts as regularization; residual decrease alone is not a guarantee of reconstruction accuracy. The Landweber spectral filter makes this progression explicit.
Learning rate 2026-11-05
A learning rate controls the size of an iterative learning update. In gradient descent, it scales a direction obtained from the gradient of a loss:
A stochastic gradient descent update uses an estimated gradient in this expression. The documented SGD algorithm makes the learning rate an explicit multiplier; momentum and other modifications add separate behavior.[ref]
A scalar example shows why size matters. For , the gradient is , so a constant learning rate gives
For nonzero initial value, convergence to zero occurs exactly when , or . With and , the values are . With , they begin : increasing the step has made the iteration diverge rather than merely learn faster.
This threshold belongs to the stated quadratic. Changing curvature changes the allowed scale: for with , the same proof gives . A numerical learning rate therefore has meaning only together with the objective and update rule.
In reinforcement learning, an incremental value estimate can instead use the update . For an illustrative , observed target and , the new estimate is . Here the learning rate weights new evidence against the previous estimate, showing the same step-size idea without requiring a gradient calculation.
A learning rate schedule can vary the rate with iteration. Report the schedule, objective scaling and other update settings when comparing results; the rate alone does not determine stability or final accuracy.
Learning rate schedule 2026-11-05
A learning rate schedule specifies how the learning rate changes during an iterative learning procedure. Its rule can depend on the update count, a training epoch count or another declared event. The rule and the clock must both be stated: changing a rate every parameter update is different from changing it after a complete pass through the training data.
A simple step-decay example is
where counts completed schedule advances from zero. The values at are , those at are , and the value at is . The StepLR implementation documents periodic multiplication by a specified factor; its example advances the scheduler after the training and validation operations.[ref]
A different schedule admits an exact convergence calculation. For gradient descent on , take . With no other update terms,
Multiplying the successive ratios gives a telescoping product:
Starting at gives . This proves convergence for the stated deterministic quadratic, rather than for every objective or noisy learning procedure.
The underlying stochastic gradient descent update also depends on its gradient estimate and any additional terms, not just the scheduled scalar.[ref] Report the initial index, update order and any external changes to the rate. The StepLR documentation explicitly permits other changes alongside its own, so the isolated step formula above assumes that no competing rule intervenes.[ref]
Nonlinear programming 2026-10-13
A nonlinear programming problem minimizes an objective over finitely many real variables subject to constraints, with at least one objective or constraint nonlinear. Parametrizing the pulse amplitudes in quantum optimal control produces such a problem. Gradient descent and related local methods exploit differentiability, but a local stationary point need not be a global optimum without additional convexity or other structure.
Optimization algorithm 2026-11-05
An optimization algorithm is a procedure for finding or approximating an optimum of an objective function subject to specified optimization constraints. Its output must be assessed through feasibility, objective quality and a justified stopping rule, rather than only the number of iterations performed.
For , gradient descent with fixed step gives . It converges to the minimizer for , oscillates without decay at when , and diverges for . This elementary example shows why an update formula needs a step-size condition. For a nonconvex objective, a small gradient can certify approximate stationarity without certifying a global minimum.
Optimizer update 2026-11-05
An optimizer update changes the adjustable parameters of an optimization problem according to an algorithm's current information. Evaluating a loss function, computing its gradient and changing the parameters are distinct operations. A general update may also change an optimizer state, such as stored information from previous steps:
Here is the supplied gradient and a learning rate. The PyTorch interface makes this separation explicit with gradient computation followed by optimizer.step().[ref]
For plain gradient descent with no extra state, . In an original scalar example, choose , and . The gradient is 2, so the update gives and lowers the loss from 2 to 1.62. Recompute the gradient at the new parameter value to obtain . Reusing the old gradient 2 instead would give 1.6. This explicitly locates where the algorithm obtains its information.[ref]
An update is not automatically an improvement: using for the first step above gives and loss 8. With gradient accumulation, several gradient calculations instead contribute before a single optimizer update; their weighting sets the effective batch size. Track the update count separately from the number of processed groups, particularly when applying a learning rate schedule.