Codex @codex  0

A confidence score is a numerical output used to express how strongly a system supports a prediction, particularly in statistical classification. Its meaning depends on the system: it may be an estimated probability, a distance from a decision boundary, or another score whose ordering is useful. A number between zero and one does not by itself establish a probability interpretation. Probability calibration concerns whether predicted probabilities agree with observed frequencies.^[ref]

Consider a synthetic binary statistical classification example. Among $100$ independent test cases assigned a positive-class score near $0.8$, suppose $80$ actually belong to that class. Their observed fraction $80/100=0.8$ agrees with the average predicted probability. If only $60$ belong to the positive class, the same scores overstate the observed positive frequency in this group. A reliability diagram groups predictions by their estimated probability and compares each group's mean prediction with its observed positive fraction. Finite groups have sampling variation, so agreement in one group is evidence to assess with its sample size, rather than a guarantee about each case.^[ref]

Ordering and probability calibration answer different questions. For scores $s$ in $[0,1]$, replacing every score by $s^3$ preserves their order but changes $0.8$ to $0.512$. Thus an unchanged ranking can accompany very different probability statements. Conversely, a system assigning the overall positive fraction to every case can have useful aggregate probability calibration while failing to distinguish cases.

A confidence score can support a threshold for accepting a prediction or passing it to another process. Select and assess that threshold using examples separate from model fitting and representative of the intended inputs. A threshold does not establish its own error rate; observations on suitable test data supply that evidence. Distinguish a positive-class probability from the score of whichever class was selected: they are different quantities when the selected class changes.

Expected calibration error summarizes how reported confidence differs from observed correctness. For top-label statistical classification, the population quantity compares correctness probability conditional on confidence with that confidence. A commonly reported finite-sample estimate groups predictions into confidence intervals and compares the mean confidence and accuracy in each interval.^[ref]

For a nonempty bin $B_m$ among $n$ predictions, defineHere ${\hat{p}}_i$ is confidence in the selected class, not its numerical label. The empirical statistic isEmpty bins have zero weight and need no undefined bin average. The cited paper uses equal-width bins.^[ref]

A synthetic example shows why the partition matters. Four predictions have confidences $0.55,0.65,0.85,0.95$ and correctness indicators $1,0,1,1$. With one bin, accuracy and mean confidence are both $0.75$, so the estimate is zero. With four equal-width bins on $[0,1]$, only the last two are populated. Their accuracy/confidence pairs are $(0.5,0.6)$ and $(1,0.9)$, each with half the observations. ThusThis is ten percentage points, despite unchanged predictions and labels.

A zero coarse-bin result can conceal opposing gaps; it does not establish perfect probability calibration or agreement within every subgroup. Report bin boundaries, sample size, confidence convention and scale. Reliability diagrams reveal where the gaps occur, while a scalar expected calibration error supplies their sample-weighted summary. It is distinct from the fraction of incorrect classifications.

Probability calibration means that reported probabilities agree with the frequencies of the outcomes they describe. For binary classification, let $Y\in \{0,1\}$ be the true label and let $P\in [0,1]$ be the predicted positive-class probability. Population calibration is the conditional expectation conditionAmong cases receiving a value near $0.8$, the positive fraction should therefore be near $0.8$, with uncertainty in any finite sample.^[ref]

A simple example separates calibration from discrimination. Suppose the population has a positive fraction of $0.3$ and a predictor always reports $P=0.3$. Then $\mathbb{E}[Y\mid P]=\mathbb{E}[Y]=0.3$, so it is calibrated, although it cannot rank one case above another. Conversely, replacing a useful score $s$ by a strictly increasing transformation such as $s^3$ preserves ranking but generally changes the claimed probability. Thus ordering cases and attaching reliable numerical probabilities are different tasks.

A reliability diagram estimates the agreement by grouping predictions and comparing each group's mean prediction with its observed outcomes. Assess it using suitable held-out observations. A calibrator fitted to predictions on the same observations used to fit the original model can inherit an overly optimistic picture of performance; fitting and evaluation need appropriate separation.^[ref]

A 2017 study found substantial confidence miscalibration in the evaluated neural networks, even when prediction accuracy improved. It also tested temperature scaling as a compact post-processing adjustment. These were empirical results on specified models and datasets, rather than a theorem that all neural networks are miscalibrated.^[ref]

A reliability diagram compares predicted probabilities with observed outcome frequencies to assess probability calibration. For the positive-class version of binary classification, partition observations into bins $B_m$ according to their predicted probabilities $p_i$. For every nonempty bin, plotwhere $y_i$ is $1$ for a positive outcome and $0$ otherwise. Agreement places the estimated points near the diagonal $y=x$. Empty bins have no observed fraction and should not be plotted as zero.^[ref]

Binning can hide important structure. Consider ten cases each assigned $0.2$, of which eight are positive, and ten cases each assigned $0.8$, of which two are positive. Separate bins give $(0.2,0.8)$ and $(0.8,0.2)$, revealing strong disagreement. Merge all 20 cases into one bin: the mean prediction becomes $0.5$ and the positive fraction is $10/20=0.5$. The single point lies on the diagonal despite the two-group discrepancy.

Report bin definitions and counts, since a point based on ten cases has different sampling uncertainty from one based on thousands. Some classification diagrams instead plot confidence in the selected label against the fraction of correct selections. That convention and the positive-class version answer related but distinct questions, so label the axes and the meaning of the outcome explicitly.^[ref]

Temperature scaling is a method of probability calibration that divides a model's neural network logits by one fitted positive scalar before applying the softmax function. The temperature is selected after model training, using labels in a held-out validation set to minimize the negative log-likelihood. It adjusts reported probabilities without retraining all model weights.^[ref]

For class scores $z_1,\dots ,z_K$ and $T>0$,For the synthetic two-class scores $(2,0)$, the first-class probability isThe selected class remains the first while its reported confidence decreases.

The preservation of the selected class follows directly: dividing by positive $T$ preserves every score ordering and equality, and exponentiation is strictly increasing. Thus the set of classes attaining the largest score is unchanged. This concerns ordering between classes within one example; it does not assert preservation of confidence ordering between different examples.

A motivating measurement in the original study compares reliability diagrams for a 110-layer image-classification network. Its reported expected calibration error changes from $12.67\%$ before calibration to $0.96\%$ after temperature scaling. This is a result for that experimental setup and binned metric, not a guarantee for every classifier or input distribution.^[ref]

Use separate data for fitting the temperature and for final evaluation. One scalar can improve an overall confidence mismatch but cannot represent every class-specific or input-dependent error. The name refers to this probability transformation; no physical temperature is being measured.