= Automatic differentiation
{wiki}
Automatic differentiation evaluates <derivatives> by applying the <chain rule> to the elementary operations executed by a numerical program. It propagates local sensitivities through the calculation; it does not estimate them by choosing a finite increment as in <numerical differentiation>.
For $f(x)=\sin(x^2)$, record $v_1=x^2$, $v_2=\sin v_1$. A forward calculation carries both values and derivatives:
$$
\dot x=1,\qquad\dot v_1=2x\dot x,\qquad\dot v_2=\cos(v_1)\dot v_1.
$$
It returns $f'(x)=2x\cos(x^2)$. More generally, seeding an input direction $r$ computes the product $J_fr$ of a <Jacobian matrix> and a <vector>. A reverse calculation starts from an output sensitivity and accumulates contributions toward the inputs; for a scalar objective it computes the full <gradient> efficiently. This is the mechanism of <backpropagation>. https://jmlr.org/papers/volume18/17-468/17-468.pdf[Automatic Differentiation in Machine Learning: a Survey, sections3.1–3.2, printed pp9–13]{ref}
The chain-rule result is exact in exact arithmetic for the differentiable operations used; a floating-point implementation still has <roundoff error>. Branches and nondifferentiable operations require care: a derivative propagated through the executed branch need not be an ordinary <derivative> of the whole program at a branch boundary. The same chain-rule machinery can differentiate <reparameterization gradients> through simulated trajectories.
Back to article page