Diagnostic reproducibility

Diagnostic reproducibility is the consistency of diagnostic classifications when the observations are interpreted again under specified conditions. Within-observer reproducibility compares the same observer at different readings; between-observer reproducibility compares different observers. The specimen set, available information, category definitions and delay between readings all affect what is being tested.

Consistency is distinct from agreement with a reference. Two observers can give the same incorrect label; one observer can change between two plausible labels even when most classifications agree with an external comparator. For a constructed binary example, suppose the reference labels all 100 test cases positive. An observer labels 90 positive at each reading, but the ten negative readings concern disjoint case sets. Each reading agrees with the reference on $90\%$, yet the two readings agree with one another on only $80\%$. These measure different comparisons.

A primary dermatopathology study assigned 240 skin biopsy cases in sets for independent interpretation twice, at least eight months apart; 187 pathologists completed the first phase. It explicitly measured within- and between-observer concordance separately from concordance with a consensus reference.^[ref] Reported repeat agreement depended on the first classification: $76.7\%$ for class I but $35.2\%$ for class II in that study. A single pooled percentage can hide variation across diagnostic categories.^[ref]

A useful repeat-reading experiment keeps track of which cases changed label and which information was available on each occasion. Blinding to the previous answer reduces simple recall; separating the readings makes the repeat task more than immediate duplication. A consensus reference must be defined as a comparator, rather than equated with direct observation of every aspect of disease biology. Neither high consistency nor a reference-agreement percentage alone measures all consequences of a diagnostic decision.