Acceleration potential 2026-10-25
A scalar potential whose negative spatial gradient is the material acceleration. For a smooth global potential, the spatial curl of the material acceleration vanishes. On a simply connected domain the converse holds under the usual smoothness assumptions. Its physical dimension is squared length divided by squared time.
A homogeneous motion has an acceleration potential exactly when is a symmetric matrix. Necessity follows because is the spatial gradient of the material acceleration, which must be a negative Hessian matrix. For sufficiency, put and take . Direct spatial differentiation gives . Equivalently is constant: its time derivative is .
AdaGrad 2026-11-05
AdaGrad is a family of adaptive gradient methods that derives update geometry from accumulated gradient information. Its diagonal form uses a separate cumulative squared-gradient history for each coordinate. John Duchi, Elad Hazan and Yoram Singer's original paper develops diagonal and full-matrix versions, including constrained and regularized settings.[ref]
Adaptive Subgradient Methods for Online Learning and Stochastic Optimization by John Duchi and others. Section1.1, printed pp2122–2124, Eq1 and Eq5
For an unconstrained diagonal specialization, start with and usewhere is the base learning rate, keeps the denominator positive, and is a gradient or suitable subgradient. The additive stabilizer is outside the square root in this declared variant. The accumulator belongs to the optimizer state and must be retained to reproduce later updates.
For an original history example, supply two-coordinate gradientsAfter 100 observations,Choose and . The effective coordinate learning rates are thenAlthough the recent supplied gradients agree, the early large first-coordinate value still strongly reduces its update scale.
Each is nondecreasing, so with fixed and the corresponding effective learning rate is nonincreasing. For a constant nonzero supplied scalar gradient , the history is and the update magnitude isThis tends to zero as the cumulative history grows. Adam instead uses a decaying squared-gradient history, giving old observations a different influence. The choice between these histories is part of the algorithm, rather than merely a different initial learning rate.
Adam 2026-11-05
Adam, named for adaptive moment estimation, is an adaptive gradient method that scales a running gradient estimate by a running squared-gradient estimate. Diederik P. Kingma and Jimmy Lei Ba presented it as a first-order method for stochastic optimization. Its state makes the next optimizer update depend on gradient history as well as the current parameters.[ref]
Adam: A Method for Stochastic Optimization by Diederik P. Kingma and Jimmy Lei Ba. Version9, printed pp1–2, Introduction and Algorithm1
Starting from , the basic minimization rule isHere is the supplied gradient, is the learning rate, , and . The Hadamard product, square root and division act coordinatewise. Adam bias correction supplies the two normalization factors. The accumulator concerns a raw second moment, rather than a centered variance.[ref]
Adam: A Method for Stochastic Optimization by Diederik P. Kingma and Jimmy Lei Ba. Algorithm1, printed p2, epsilon outside the square root
For an original scalar calculation, choose , , and , and supply , . The first step hasThe second hasEqual and opposite supplied gradients have not undone one another: the running histories give the second step a different magnitude. Plain gradient descent with the same constant learning rate would return to zero for this sequence.
The calculation specifies the recurrence, rather than a convergence guarantee for an unspecified objective function. An optimization checkpoint needs these histories and the step count to reproduce subsequent steps. AdamW adds decoupled weight decay; feeding a penalty gradient into Adam instead implements coupled L2 regularization and changes the histories.
AdamW 2026-11-05
AdamW combines the adaptive update of Adam with decoupled weight decay. It maintains exponentially weighted histories of the supplied data gradient and its componentwise square, while applying parameter shrinkage separately. Ilya Loshchilov and Frank Hutter presented the method in their work on decoupling decay from gradient-based regularization.[ref]
Decoupled Weight Decay Regularization by Ilya Loshchilov and Frank Hutter. Version3, Algorithm2, AdamW
For minimization without optional algorithm variants, initialize . In the convention documented by PyTorch, step usesThe product is a Hadamard product; square roots and division are also coordinatewise. Assume , and the stated learning rate . The decay coefficient is . This explicitly fixes epsilon placement and the decay convention.[ref]
AdamW in the PyTorch documentation. PyTorch2.15, algorithm with amsgrad and maximize disabled
The denominators and perform Adam bias correction for zero-initialized histories. For a constant supplied scalar gradient , the corresponding geometric sums give and . Their corrected values are therefore and . The squared-gradient quantity concerns a raw second moment, rather than a centered variance: a constant gradient of 2 has square 4 even though its variance is zero.
For an original first-step example, choose , , , , , and . ThenandThe constants are chosen for transparent arithmetic. Feeding into the same adaptive formulas instead, with no separate shrinkage, would give approximately . That coupled L2 regularization also changes the stored histories. Resuming AdamW therefore requires the relevant optimizer state and step count as well as the parameter values and settings.
Adaptive gradient method 2026-11-05
An adaptive gradient method uses information from supplied gradients to adjust subsequent optimizer updates, often with a different scale for each coordinate. A useful diagonal form iswhere is the current gradient, is a scalar learning rate and is a positive diagonal matrix computed from gradient history. A learning rate schedule can still control independently of the coordinate adaptation. The broader family includes AdaGrad, Adam and AdamW.
The 2011 AdaGrad paper by John Duchi, Elad Hazan and Yoram Singer formalized adaptive geometry from accumulated gradient information, including diagonal and full-matrix constructions.[ref]
Adaptive Subgradient Methods for Online Learning and Stochastic Optimization by John Duchi and others. Section1.1, printed pp2122–2124, Eq1 and Eq5
An unconstrained diagonal specialization maintainsHere specifies additive stabilization outside the square root. The accumulator is part of the optimizer state, rather than determined by the current parameters alone.For an original example, start at and supply , with and . Both accumulated entries are positive, so the denominators are defined. The first step givesPlain gradient descent with the same scalar learning rate would instead give . If the next supplied gradient is again , then and each coordinate's additional movement is . These are explicit supplied-gradient calculations, rather than a convergence claim.
In this cumulative version, an accumulated squared-gradient entry never decreases, so its effective coordinate learning rate does not increase. Other adaptive gradient methods retain decaying averages instead, changing how strongly old gradients influence later updates. Adaptation therefore specifies both the state recurrence and how that state transforms the gradient; the word “adaptive” alone does not determine either.
Adiabatic initial conditions Created 2026-09-24 Updated 2026-11-05
Adiabatic initial conditions perturb all relevant local physical quantities along the same background time trajectory. For quantities with nonzero background time derivatives, their perturbations satisfy . For separately conserved barotropic fluids, cosmological energy conservation then givesFor example, pressureless matter and radiation in cosmology obey . Their relative particle-number ratio is unperturbed at leading order.
The density condition tests relative entropy between components. In a general nonbarotropic component it does not by itself eliminate intrinsic entropy, measured by . The stronger statement that every physical variable follows the same time shift includes the pressure and any independent field-velocity data. A single scalar species alone does not make the density-ratio test sufficient, since that test is then vacuous.
For vanishing total non-adiabatic pressure perturbation and negligible regular gradient terms, uniform-density curvature is conserved on superhorizon scales.[ref]
A new approach to the evolution of cosmological perturbations on large scales by David Wands and others. Equations (18)–(19), (29), and Section V.A
Relating this to conserved comoving curvature perturbation also requires the appropriate regular adiabatic mode; a non-attractor phase such as ultra-slow-roll inflation needs separate analysis. The ordinary growing adiabatic photon mode selects coherent acoustic phases, explaining the adiabatic acoustic-peak spacing. Alternating proximal-gradient operator Created 2026-10-06 Updated 2026-11-05
Let be a differentiable convex function whose full gradient is -Lipschitz continuous, with . The alternating update first solves the implicit step and then the explicit step:The implicit step is the proximal operator of , so it exists uniquely. The gradient bound must control the full joint variable; smoothness alone supplies no global constant .
For two updates, let be the difference of their mixed points and the corresponding full gradient difference. The input and output differences are and . The mixed-point norm identity for alternating updates and Baillon–Haddad theorem giveHence is a -averaged operator for . Its fixed points have both block gradients zero and are exactly the global minimizers of . If a minimizer exists, Browder convergence theorem for averaged operators gives norm convergence of iteration in the stated finite-dimensional setting.
Angle between intersecting regular surfaces 2026-10-20
For smooth level surfaces , meeting at a point where both gradients are nonzero, their acute angle is the angle between two planes tangent to the surfaces there. Each gradient is a normal vector to its tangent plane, proving the displayed formula. Vanishing gradients require further analysis because the level-set equation then need not specify a unique tangent plane.
Anisotropic Gaussian smectic scaling Created 2026-10-14 Updated 2026-11-05
For a Gaussian kernel , keeping both gradient coefficients fixed gives and . The volume rescales by . Fourier-field normalization is ; the real-space normalization instead divides by . The thermal coefficient scales by and the uniform source by . Thus , and the formal Gaussian free-energy powers are , .
Annular Caccioppoli inequality Created 2026-10-06 Updated 2026-10-14
For a weak solution of a uniformly elliptic homogeneous divergence-form elliptic operator, testing with and choosing the gradient of the cutoff function in gives . The arbitrary constant can be chosen as an annular average, which makes the estimate suitable for a hole-filling argument.
Automatic differentiation Created 2026-10-06 Updated 2026-11-05
Automatic differentiation evaluates derivatives by applying the chain rule to the elementary operations executed by a numerical program. It propagates local sensitivities through the calculation; it does not estimate them by choosing a finite increment as in numerical differentiation.
For , record , . A forward calculation carries both values and derivatives:It returns . More generally, seeding an input direction computes the product of a Jacobian matrix and a vector. A reverse calculation starts from an output sensitivity and accumulates contributions toward the inputs; for a scalar objective it computes the full gradient efficiently. This is the mechanism of backpropagation. [ref]
https://jmlr.org/papers/volume18/17-468/17-468.pdf. Automatic Differentiation in Machine Learning: a Survey, sections3.1–3.2, printed pp9–13
The chain-rule result is exact in exact arithmetic for the differentiable operations used; a floating-point implementation still has roundoff error. Branches and nondifferentiable operations require care: a derivative propagated through the executed branch need not be an ordinary derivative of the whole program at a branch boundary. The same chain-rule machinery can differentiate reparameterization gradients through simulated trajectories.
For a flux and a weak solution of , discrete integration by parts and the fundamental theorem of calculus along a line segment show that solves , with and . Coercive quadratic-form bounds and operator norm bounds for on all segments between these gradients pass to . This produces a divergence-form elliptic operator without initially assuming second partial derivatives of .
Averaged stationarity bound for stochastic gradient descent Created 2026-10-25 Updated 2026-11-05
Suppose a fixed-step size stochastic gradient descent iteration satisfiesand has a global minimizer . Taking expectations and forming a telescoping sum proves the displayed bound with and deterministic . The Cauchy-Schwarz inequality then bounds the averaged, or the best, expected gradient Euclidean norm by . This is a guarantee of stationarity in the indicated senses, rather than a last-iterate convergence assertion. The additive is a stochastic gradient noise floor.
Target: real-analysis.bigb, under Strongly convex function. Axel D. Becke Created 2026-10-20 Updated 2026-11-05
Axel D. Becke develops practical approximations for density functional theory. His 1988 exchange functional uses the local electron density and its gradient to improve on a purely local density approximation. Its construction imposes the proper long-distance behavior of the exchange-energy density and fits one parameter using atomic exchange energies. This supplies a concrete example of building a useful approximate energy functional from physical constraints and reference calculations. The asymptotic statement concerns an energy density, and does not automatically establish the asymptotic behavior of its functional derivative.[ref]
Density-functional exchange-energy approximation with correct asymptotic behavior by Axel D. Becke. Physical Review A 38, 3098 (1988), abstract and exchange-functional construction
Portrait of Axel D. Becke
. External portrait hosted by Dalhousie University, Department of Chemistry. Source. Backpropagation through time 2026-10-25
Apply the chain rule to a recurrent network unrolled over its input sequence, adding contributions to parameters reused at multiple time steps. Products of state Jacobians convey effects from earlier states to later losses. This can compute exact gradients for the chosen finite unrolling, while physiological availability of the same history and error signals is a separate question.
Barotropic vorticity transport Created 2026-10-05 Updated 2026-11-05
For an inviscid barotropic fluid with only potential body forces, the vorticity obeysThe pressure force is a gradient because . A spatially uniform specific entropy makes a fixed-composition fluid barotropic. Merely having does not exclude baroclinic vorticity generation from spatial entropy gradients.
Batch size 2026-11-05
Batch size counts the examples combined in a mini-batch. In the simple stochastic gradient descent loop where each group produces one parameter update, it controls how many examples contribute to that update. It differs from a training epoch, which counts a pass over the available training data.[ref]
Optimizing Model Parameters in the PyTorch tutorials. Hyperparameters and Optimization Loop
For an original counting example, partition ten examples with nominal batch size and retain the remainder. The actual sizes are , giving updates per training epoch. Discarding the remainder instead gives two updates and uses only eight examples. More generally the retained-remainder update count is for examples, provided there is one update per group.
Batch size also changes gradient sampling noise. If independently sampled component gradients at a fixed parameter have scalar variance , their sample mean hasFor an original exact example, suppose one component gradient is equally likely to be 0 or 2. Its mean is 1 and variance is 1. A two-example mini-batch gradient takes values with probabilities , giving variance . This calculation holds parameters fixed and assumes independent sampling; sampling without replacement has a finite population correction.
With gradient accumulation, several groups may contribute before an update. Two full four-example groups then involve eight examples per update if their gradients are normalized consistently. State whether a reported size counts the processed group or all examples contributing to an update. Lower sampling variance alone does not establish faster training for a fixed computation budget.
Birefringent walk-off 2026-10-14
The extraordinary wave's Poynting vector can differ in direction from its wavevector. The latter specifies wave phase propagation, while the former specifies energy flux. This angular separation is walk-off.[ref]
In the lossless, nondispersive model above, take the gradient of to obtain the group velocity, parallel to the Poynting vector. If its angle to the optic axis is , this gives . Consequently the extraordinary wave generally carries energy away from its wavevector direction. An internal wavevector normal to a slab face therefore does not guarantee a normal energy path. Birefringent walk-off vanishes at and ; the latter direction still has unequal refractive indices. This is a useful waveplate geometry: overlapping components can acquire phase retardation without spatial splitting. A thin specimen, insufficient imaging resolution, or excitation of only one linear polarization can also hide a double image, for different reasons.
Block coordinate gradient 2026-10-25
The orthogonal projection of the gradient onto the coordinate subspace indexed by . It is kept as a vector in the original Euclidean space, with zero coordinates outside the block. For a singleton block it is ; for the full block it is .
Target: gradient-descent.bigb, under Lipschitz gradient.