A differential test runs implementations intended to realize the same contract on matched inputs and compares their observable results. Disagreement reveals an inconsistency without requiring a precomputed answer. Hypothesis explicitly supports comparisons using either a trusted reference or several implementations none of which is fully trusted.[ref]
Integrations Reference in the Hypothesis documentation. equivalent(), reference-oracle and differential-testing examples
For example, compare a new sorting algorithm with a simple reference on lists containing duplicates, negative values and empty input. Compare results under the specified ordering contract; unspecified differences are not failures. Agreement alone cannot exclude a shared software bug. Floating-point arithmetic may require a justified numerical tolerance, and comparison of failures as well as successful outputs requires an explicit error contract.
Articles by others on the same topic
There are currently no matching articles.