Within-observer agreement versus reference agreement ($A_{\mathrm{within}},A_{\mathrm{ref}}$)

Within-observer agreement versus reference agreement distinguishes consistency of an observer's repeated labels from concordance with a separately defined reference. The first asks whether the observer repeats a decision; the second asks whether that decision matches the comparison standard. In statistical classification, neither quantity can replace the other.

For $n$ cases, let $X_{i1}$ and $X_{i2}$ be one observer's labels at two readings, and $Y_i$ the reference label. Define the two empirical agreement fractionsAn original binary example has 100 cases, half reference-positive and half reference-negative. An observer reverses every reference label and repeats that reversal on the second reading. Then $A_{\mathrm{within}}=1$ while $A_{\mathrm{ref}}=0$. Perfect repeat agreement can coexist with complete reference disagreement. An observer always assigning the negative label also has perfect repeat agreement, but only 50 percent reference agreement in this balanced example.

A confusion matrix shows which categories contribute to disagreement, whereas a single overall percentage can conceal systematic shifts or an imbalanced case mix. Reference agreement is classification accuracy relative to that reference, not proof that the reference is infallible. A consensus panel, a later outcome and an independent molecular test answer different validation questions.

A primary 2017 study made this distinction experimentally by having pathologists interpret skin-biopsy sets twice, at least eight months apart, and comparing their interpretations with an experienced consensus reference. The study started with 240 cases and reported reproducibility separately from reference concordance.^[ref] The delayed repeat reduced immediate recall, but did not transform repeated judgments into independent biological truths.

A useful evaluation preserves case identity, reading order and the exact reference procedure. If a machine learning classifier learns an observer's systematically shifted labels, it may reproduce that observer well while disagreeing with the external reference. Reporting both comparisons makes that limitation visible. Uncertainty estimates should also account for repeated cases and multiple readings rather than counting every label as an independent subject.