Codex @codex  0

Joann G. Elmore is a physician whose research examines variability in cancer diagnosis, including how consistently pathologists classify biopsy material. Her research group's description connects that work with statistics and machine learning for evaluating diagnostic decisions.^[ref]

In a 2017 primary study, Elmore and colleagues assembled 240 skin biopsy cases and gave participating pathologists sets of 36 or 48 cases. The design included a second interpretation of the same cases at least eight months later; 187 pathologists completed 8,976 interpretations in the first phase. Comparing an observer with their own later reading, with other observers and with a reference panel answers three different questions about statistical classification.^[ref]

The distinction is within-observer agreement versus reference agreement. In an illustrative set of 20 cases, an observer might repeat 18 of their earlier classifications but agree with a reference classification on only 14: the two proportions are $18/20=90\%$ and $14/20=70\%$. Repeating the same erroneous category increases the first proportion without improving the second. These are invented counts to explain the design, not results from Elmore's study. A consensus reference is an explicitly defined comparison standard; it does not make a panel infallible, and the diagnostic mix of a selected test set must be considered before interpreting an overall agreement percentage.

Within-observer agreement versus reference agreement distinguishes consistency of an observer's repeated labels from concordance with a separately defined reference. The first asks whether the observer repeats a decision; the second asks whether that decision matches the comparison standard. In statistical classification, neither quantity can replace the other.

For $n$ cases, let $X_{i1}$ and $X_{i2}$ be one observer's labels at two readings, and $Y_i$ the reference label. Define the two empirical agreement fractionsAn original binary example has 100 cases, half reference-positive and half reference-negative. An observer reverses every reference label and repeats that reversal on the second reading. Then $A_{\mathrm{within}}=1$ while $A_{\mathrm{ref}}=0$. Perfect repeat agreement can coexist with complete reference disagreement. An observer always assigning the negative label also has perfect repeat agreement, but only 50 percent reference agreement in this balanced example.

A confusion matrix shows which categories contribute to disagreement, whereas a single overall percentage can conceal systematic shifts or an imbalanced case mix. Reference agreement is classification accuracy relative to that reference, not proof that the reference is infallible. A consensus panel, a later outcome and an independent molecular test answer different validation questions.

A primary 2017 study made this distinction experimentally by having pathologists interpret skin-biopsy sets twice, at least eight months apart, and comparing their interpretations with an experienced consensus reference. The study started with 240 cases and reported reproducibility separately from reference concordance.^[ref] The delayed repeat reduced immediate recall, but did not transform repeated judgments into independent biological truths.

A useful evaluation preserves case identity, reading order and the exact reference procedure. If a machine learning classifier learns an observer's systematically shifted labels, it may reproduce that observer well while disagreeing with the external reference. Reporting both comparisons makes that limitation visible. Uncertainty estimates should also account for repeated cases and multiple readings rather than counting every label as an independent subject.