 Automatic differentiation

Automatic differentiation evaluates derivatives by applying the chain rule to the elementary operations executed by a numerical program. It propagates local sensitivities through the calculation; it does not estimate them by choosing a finite increment as in numerical differentiation.

For $f(x)=\sin (x^2)$, record $v_1=x^2$, $v_2=\sin v_1$. A forward calculation carries both values and derivatives:It returns $f^{\prime }(x)=2x\cos (x^2)$. More generally, seeding an input direction $r$ computes the product $J_{f}r$ of a Jacobian matrix and a vector. A reverse calculation starts from an output sensitivity and accumulates contributions toward the inputs; for a scalar objective it computes the full gradient efficiently. This is the mechanism of backpropagation. ^[ref]

The chain-rule result is exact in exact arithmetic for the differentiable operations used; a floating-point implementation still has roundoff error. Branches and nondifferentiable operations require care: a derivative propagated through the executed branch need not be an ordinary derivative of the whole program at a branch boundary. The same chain-rule machinery can differentiate reparameterization gradients through simulated trajectories.

Automatic differentiation (AD) is a computational technique used to evaluate the derivative of a function specified by a computer program. AD is particularly useful in various fields including machine learning, optimization, and scientific computing because it allows for efficient and accurate computation of derivatives, which is crucial for gradient-based optimization methods.