Automatic differentiation Created 2026-10-06 Updated 2026-11-05
Automatic differentiation evaluates derivatives by applying the chain rule to the elementary operations executed by a numerical program. It propagates local sensitivities through the calculation; it does not estimate them by choosing a finite increment as in numerical differentiation.
For , record , . A forward calculation carries both values and derivatives:It returns . More generally, seeding an input direction computes the product of a Jacobian matrix and a vector. A reverse calculation starts from an output sensitivity and accumulates contributions toward the inputs; for a scalar objective it computes the full gradient efficiently. This is the mechanism of backpropagation. [ref]
https://jmlr.org/papers/volume18/17-468/17-468.pdf. Automatic Differentiation in Machine Learning: a Survey, sections3.1–3.2, printed pp9–13
The chain-rule result is exact in exact arithmetic for the differentiable operations used; a floating-point implementation still has roundoff error. Branches and nondifferentiable operations require care: a derivative propagated through the executed branch need not be an ordinary derivative of the whole program at a branch boundary. The same chain-rule machinery can differentiate reparameterization gradients through simulated trajectories.
Ill-posedness of numerical differentiation 2026-10-31
Recovering a derivative from values controlled only in the uniform norm is an ill-posed problem: the differentiation map is not continuous in those norms. On , take . Then , while has uniform norm one for every . Arbitrarily small value perturbations can therefore conceal nonvanishing derivative changes.
Numerical differentiation can still be useful when additional smoothness bounds, a suitable sampling scale, or a regularization method control the admissible changes. The basic noise amplification by differentiation of a linear interpolant gives a finite-data version: endpoint uncertainty of size produces derivative uncertainty of order at spacing .
For linear interpolation at zero and , an error in the second sampled value, with the first exact, changes the linear interpolant by . On , its uniform norm is exactly , whereas the derivative change is the constant . With independent absolute bounds on both samples, the value change remains at most and the derivative change is at most , attained for opposite endpoint errors.
Thus bounded absolute sample noise can remain bounded under value interpolation while becoming unbounded under numerical differentiation as . Fixed nonzero sample noise also prevents exact convergence of the value approximation; stability against perturbations and convergence to exact data are different assertions. For a second-sample error with fixed , value convergence requires , while derivative convergence requires in little o notation, in addition to vanishing truncation error.
Numerical differentiation 2026-10-31
Numerical differentiation estimates derivatives from finitely many function values, often by differentiating an interpolation polynomial or using a finite difference. Its error combines truncation error from the formula with errors already present in the sampled values and additional roundoff error. Decreasing the sample spacing can decrease truncation error while increasing the effect of uncertain values. This tradeoff is explained by the ill-posedness of numerical differentiation when only an absolute value-error bound is available.