= Expected calibration error
{title2=$\widehat{\mathrm{ECE}}=\sum_m(|B_m|/n)|\operatorname{acc}(B_m)-\operatorname{conf}(B_m)|$}
<Expected calibration error> summarizes how reported confidence differs from observed correctness. For top-label <statistical classification>, the population quantity compares correctness probability conditional on confidence with that confidence. A commonly reported finite-sample estimate groups predictions into confidence intervals and compares the <mean> confidence and accuracy in each interval.<On Calibration of Modern Neural Networks by Chuan Guo and others>[§2, Eqs. (2)–(3), population calibration gap and binned estimate]{ref}
For a nonempty bin $B_m$ among $n$ predictions, define
$$
\operatorname{acc}(B_m)=\frac{1}{|B_m|}\sum_{i\in B_m}\mathbf1\{\hat y_i=y_i\},
\qquad
\operatorname{conf}(B_m)=\frac{1}{|B_m|}\sum_{i\in B_m}\hat p_i.
$$
Here $\hat p_i$ is confidence in the selected class, not its numerical label. The empirical statistic is
$$
\widehat{\mathrm{ECE}}=
\sum_{m:B_m\ne\varnothing}\frac{|B_m|}{n}
\left|\operatorname{acc}(B_m)-\operatorname{conf}(B_m)\right|.
$$
Empty bins have zero weight and need no undefined bin average. The cited paper uses equal-width bins.<On Calibration of Modern Neural Networks by Chuan Guo and others>[§2, bin definitions and Eq. (3), printed pp.1322–1323]{ref}
A synthetic example shows why the partition matters. Four predictions have confidences $0.55,0.65,0.85,0.95$ and correctness indicators $1,0,1,1$. With one bin, accuracy and mean confidence are both $0.75$, so the estimate is zero. With four equal-width bins on $[0,1]$, only the last two are populated. Their accuracy/confidence pairs are $(0.5,0.6)$ and $(1,0.9)$, each with half the observations. Thus
$$
\widehat{\mathrm{ECE}}=\tfrac12|0.5-0.6|+\tfrac12|1-0.9|=0.1.
$$
This is ten percentage points, despite unchanged predictions and labels.
A zero coarse-bin result can conceal opposing gaps; it does not establish perfect <probability calibration> or agreement within every subgroup. Report bin boundaries, sample size, confidence convention and scale. <Reliability diagrams> reveal where the gaps occur, while a scalar <expected calibration error> supplies their sample-weighted summary. It is distinct from the fraction of incorrect classifications.
Back to article page
