Software testing

Software testing executes a program under specified conditions and compares its behaviour with an expected result or property. A unit test may check one function; an integration test checks interacting parts; a regression test guards against recurrence of a known failure. For a sorting routine, useful properties include ordered output and preservation of the input multiset. Passing finitely many tests cannot generally prove correctness for every input. The choice of cases and the quality of the test oracle determine what evidence a test actually supplies.

A test case specifies conditions, inputs, an operation to perform and the expected result or acceptance property for software testing. Its test oracle must distinguish success from failure.^[ref] Merely executing a computer program without checking an outcome supplies weaker evidence.

For example, a sorting algorithm given 3,1,3 should return 1,3,3. A stronger test oracle checks both sortedness and preservation of the input multiset; checking only sortedness would accept an empty result. The test case also needs a policy for equal elements if their original order matters. Explicit preconditions and expected outcomes make a failure reproducible and help a later reader understand which contract the test case exercises.

A differential test runs implementations intended to realize the same contract on matched inputs and compares their observable results. Disagreement reveals an inconsistency without requiring a precomputed answer. Hypothesis explicitly supports comparisons using either a trusted reference or several implementations none of which is fully trusted.^[ref]

For example, compare a new sorting algorithm with a simple reference on lists containing duplicates, negative values and empty input. Compare results under the specified ordering contract; unspecified differences are not failures. Agreement alone cannot exclude a shared software bug. Floating-point arithmetic may require a justified numerical tolerance, and comparison of failures as well as successful outputs requires an explicit error contract.

A software regression is a loss of previously working behavior after a change. The behavior may concern outputs, accepted inputs, performance, reliability or compatibility. A regression is defined against an intended contract and earlier behavior, rather than every difference from an old implementation.

For example, a parser update can correctly handle a new syntax form while accidentally rejecting an older supported form. A regression test records a case and its expected behavior so that the failure is detected on later changes. Another change can leave outputs unchanged but make a common workload much slower; that requires a performance criterion and representative workload. Passing an existing suite only supplies evidence for its covered cases and oracles, so fixing a discovered regression often includes adding a focused new test.

A test oracle determines whether an observed test result is acceptable. It can be an explicitly expected output, a mathematical property, a trusted reference implementation or another justified source of comparison. Generating many inputs without an oracle does not by itself establish whether the results are correct.

For a sorting routine, an oracle can check both sortedness and preservation of the input multiset. Returning an empty sequence passes a sortedness check but fails preservation for a nonempty input. A differential test instead compares implementations; a disagreement exposes a problem without necessarily identifying which implementation is wrong. Hypothesis documents both reference-oracle comparisons and differential comparisons in its equivalence-test generator.^[ref]

Test a conversion together with its intended inverse. Equality of representations may need normalization, while equality of numerical values can be limited by floating-point rounding or truncation. Distinct strings can encode the same value, so a value round trip and a character-for-character round trip are different properties.

An independent oracle is still needed: the wrong functions $f(x)=x+1$ and $g(y)=y-1$ satisfy both composition identities on integers, although neither implements an intended identity conversion. Thus passing inverse-consistency tests does not alone establish the specification of either conversion.

Select tests by dividing an input domain into classes expected to exercise the same relevant behavior, then choosing representatives, including boundaries. Integer-only and point-containing inputs can exercise different paths in a positional-number parser. The classes are part of the test design, not a proof that one representative detects every possible defect in its class. Invalid-format classes need an explicit validation contract before an expected rejection can be asserted.

Choose tests using knowledge of internal structure, such as branches, paths or representation cases. Covering all branches is still not generally a proof of correctness for every input. This selection criterion can be combined with specification-based tests within test-driven development.

Choose tests from externally specified input/output behaviour without using the implementation's internal control structure as the selection criterion. Boundary cases can be black-box tests when the boundary comes from the contract. Test authorship by the implementer does not alone determine this classification.

A development cycle that starts with a small expected-to-fail behavioural test, adds implementation until it passes, and refactors while preserving the tests. Its workflow is distinct from the source of test cases: public-contract tests are black-box tests, while implementation-informed branch cases can be white-box tests. The practice does not by itself prove completeness of the test suite.

Continuous integration is a software engineering practice in which collaborators frequently combine small changes into a shared main line, conventionally at least daily, and check each integration through an automated software build and software testing. Fast feedback makes incompatible assumptions visible while the changes are still small. A server that tests isolated development branches contributes automation; frequent integration into the shared main line is the defining collaboration practice.^[ref]

For example, one developer changes a parser to return a new data structure, while another changes code that consumes the result. Each component can pass its own isolated unit tests while their combination fails. Integration testing of the shared revision exercises that interface and exposes the mismatch. The build result should identify the tested source revision so that a failure can be reproduced.

An automated software build turns declared source inputs and dependencies into outputs through repeatable tool steps, such as code generation, compilation, linking and packaging. A build description defines those steps and their dependencies; a build system decides their order and may reuse outputs whose relevant inputs have not changed.

For example, changing a header can require recompiling several source files before relinking a computer program. Omitting that dependency can produce a stale executable even when every individual compiler invocation succeeds. In continuous integration, the automated build provides a repeatable check of the integrated revision, often combined with tests.^[ref] Repeatability requires identifying tool versions, generated inputs and configuration as well as the main source files.

A build system executes declared transformations from source files and other inputs to build outputs, respecting their dependency graph. Common steps include code generation, compilation, linking and packaging. CMake describes builds through targets such as executables and software libraries, with dependencies between them.^[ref]

For example, main.o and math.o must exist before they can be linked into an executable. If only the source for math.o changes, an incremental build can reuse main.o, rebuild math.o and relink. Reuse is valid only when every relevant requirement has been tracked: a changed header, tool option or generated input can invalidate an apparently unchanged output. Correct dependency declarations are part of build correctness.

Testing relations between outputs for related inputs when an exact reference answer is unavailable. The expected relation must be valid for the scientific problem.

Generating cases and checking general properties such as conservation, symmetry or bounds rather than only selected input-output pairs.

Checking whether a software change alters previously established behavior. A reference output must first be justified rather than merely copied from old software.

Testing interactions between software components, including data formats, units and ordering.

Testing a small software component against a specified expected behavior.

Test coverage measures which selected program elements a test run exercises. Statement coverage counts executed statements; branch coverage counts executed control-flow alternatives. Exercising both sides of a conditional can reveal gaps missed by statement coverage, but even full branch coverage need not exercise all paths or meaningful value combinations. Coverage records execution, not whether assertions checked the right behaviour. A suite that calls every function and verifies nothing can have impressive coverage while failing to detect wrong results.