Confidence score
= Confidence score
A <confidence score> is a numerical output used to express how strongly a system supports a prediction, particularly in <statistical classification>. Its meaning depends on the system: it may be an estimated <probability>, a distance from a decision boundary, or another score whose ordering is useful. A number between zero and one does not by itself establish a <probability> interpretation. <Probability calibration> concerns whether predicted <probabilities> agree with observed frequencies.<Probability calibration in the scikit-learn documentation>[§1.16, opening explanation of calibrated classifiers and predict_proba versus decision_function]{ref}
Consider a synthetic binary <statistical classification> example. Among $100$ independent test cases assigned a positive-class score near $0.8$, suppose $80$ actually belong to that class. Their observed fraction $80/100=0.8$ agrees with the average predicted <probability>. If only $60$ belong to the positive class, the same scores overstate the observed positive frequency in this group. A <reliability diagram> groups predictions by their estimated <probability> and compares each group's mean prediction with its observed positive fraction. Finite groups have sampling variation, so agreement in one group is evidence to assess with its sample size, rather than a guarantee about each case.<Probability calibration in the scikit-learn documentation>[§1.16.1, Calibration curves: binning, average predicted probabilities and fractions of positives]{ref}
Ordering and <probability calibration> answer different questions. For scores $s$ in $[0,1]$, replacing every score by $s^3$ preserves their order but changes $0.8$ to $0.512$. Thus an unchanged ranking can accompany very different <probability> statements. Conversely, a system assigning the overall positive fraction to every case can have useful aggregate <probability calibration> while failing to distinguish cases.
A <confidence score> can support a threshold for accepting a prediction or passing it to another process. Select and assess that threshold using examples separate from model fitting and representative of the intended inputs. A threshold does not establish its own error rate; observations on suitable test data supply that evidence. Distinguish a positive-class <probability> from the score of whichever class was selected: they are different quantities when the selected class changes.