Round-off error

Round-off error arises when numbers or arithmetic results are rounded to a finite set of representable values. Its scale is associated with machine epsilon, and its effects depend on subsequent operations. Even a formula with zero truncation error can lose information. In decimal arithmetic rounded to three significant digits, $1.234$ becomes $1.23$ and $1.233$ becomes $1.23$: subtracting the stored numbers gives zero although the exact difference is $0.001$. The subtraction exposes errors already present in its nearly equal operands.

This example also has a large condition number: independently perturbing the two inputs of $a-b$ can produce a relative amplification $(|a|+|b|)/|a-b|=2467$. High sensitivity belongs to the input-output problem; numerical stability concerns the arithmetic used to solve it. Conversely, on a common binary system with 53 significant bits and rounding to nearest, $(10^{16}+1)-10^{16}$ evaluates to zero while $(10^{16}-10^{16})+1$ evaluates to one. Exact arithmetic makes both one: changing summation order changes the accumulated roundoff error. MIT's summation lecture explains why numerical stability and the condition number must be assessed separately.^[ref]

Near a smooth local minimum $x_{\ast }$ with $f^{\prime }(x_{\ast })=0$ and $f^{\prime \prime }(x_{\ast })>0$, Taylor's theorem givesIf evaluated function values have absolute uncertainty of order $\delta f$, differences on that scale cannot reliably identify smaller displacements than order $\sqrt{\delta f/f^{\prime \prime }(x_{\ast })}$. In ordinary floating-point arithmetic, when $f(x_{\ast })\ne 0$ and evaluations have relative error of order $ϵ$, this becomesFor unit-scale values and curvature, this is the square root of machine epsilon, rather than machine epsilon itself. It explains why a derivative-free golden-section search can lose reliable comparisons while candidate positions remain representably different.^[ref]

This is a precision limit of the evaluated values, not an invariant property of the minimizer. Accurately subtracting a known constant, obtaining values at higher precision, or locating a zero of an accurately evaluated derivative can change the attainable position accuracy. A zero minimum with relative error controlled on its small values also needs separate analysis.

Subtracting nearly equal approximate quantities can lose significant digits because the absolute input errors become large relative to the small difference. For example, the positive polynomial root of $a{z}^2+z-y=0$ can be evaluated as $2y/(1+\sqrt{1+4ay})$, avoiding the subtraction in $(-1+\sqrt{1+4ay})/(2a)$ when $ay$ is small.