= Probability calibration
<Probability calibration> means that reported <probabilities> agree with the frequencies of the outcomes they describe. For <binary classification>, let $Y\in\{0,1\}$ be the true label and let $P\in[0,1]$ be the predicted positive-class <probability>. Population calibration is the <conditional expectation> condition
$$
\mathbb E[Y\mid P]=P.
$$
Among cases receiving a value near $0.8$, the positive fraction should therefore be near $0.8$, with uncertainty in any finite sample.<Probability calibration in the scikit-learn documentation>[Opening definition and section 1.16.1: binary probabilities and observed positive fractions]{ref}
A simple example separates calibration from discrimination. Suppose the population has a positive fraction of $0.3$ and a predictor always reports $P=0.3$. Then $\mathbb E[Y\mid P]=\mathbb E[Y]=0.3$, so it is calibrated, although it cannot rank one case above another. Conversely, replacing a useful score $s$ by a strictly increasing transformation such as $s^3$ preserves ranking but generally changes the claimed <probability>. Thus ordering cases and attaching reliable numerical <probabilities> are different tasks.
A <reliability diagram> estimates the agreement by grouping predictions and comparing each group's mean prediction with its observed outcomes. Assess it using suitable held-out observations. A calibrator fitted to predictions on the same observations used to fit the original model can inherit an overly optimistic picture of performance; fitting and evaluation need appropriate separation.<Probability calibration in the scikit-learn documentation>[Sections 1.16.1–1.16.2: reliability diagrams and independent calibration data]{ref}
A 2017 study found substantial confidence miscalibration in the evaluated <neural networks>, even when prediction accuracy improved. It also tested <temperature scaling> as a compact post-processing adjustment. These were empirical results on specified models and datasets, rather than a theorem that all <neural networks> are miscalibrated.<On Calibration of Modern Neural Networks by Chuan Guo and others>[Figure 1 and sections 3–4: experimental comparisons and calibration methods]{ref}
Back to article page
