 Differential test

A differential test runs implementations intended to realize the same contract on matched inputs and compares their observable results. Disagreement reveals an inconsistency without requiring a precomputed answer. Hypothesis explicitly supports comparisons using either a trusted reference or several implementations none of which is fully trusted.^[ref]

For example, compare a new sorting algorithm with a simple reference on lists containing duplicates, negative values and empty input. Compare results under the specified ordering contract; unspecified differences are not failures. Agreement alone cannot exclude a shared software bug. Floating-point arithmetic may require a justified numerical tolerance, and comparison of failures as well as successful outputs requires an explicit error contract.