= Neural network logit
{title2=$p_k=e^{z_k}/\sum_j e^{z_j}$}
A <neural network logit> is a real-valued class score produced before a probability transformation such as the <softmax function>. The vector of scores determines the class <probabilities> jointly; an isolated score is not itself a probability. Negative scores and scores larger than one are permitted.<On Calibration of Modern Neural Networks by Chuan Guo and others>[§4.2, paragraph before Eq. (8), logits before the softmax layer]{ref}
For scores $z_1,\ldots,z_K$, the <softmax function> gives
$$
p_k=\frac{e^{z_k}}{\sum_{j=1}^K e^{z_j}}.
$$
Adding one common scalar $a$ to every coordinate leaves all probabilities unchanged:
$$
\frac{e^{z_k+a}}{\sum_j e^{z_j+a}}
=\frac{e^a e^{z_k}}{e^a\sum_j e^{z_j}}
=p_k.
$$
Consequently the absolute offset of a logit vector cannot be recovered from these probabilities.
For example, scores $(2,0)$ and $(7,5)$ both give first-class probability $e^2/(e^2+1)\simeq0.8808$. The difference between the two scores, rather than the first coordinate alone, determines this binary probability. Indeed,
$$
\ln\frac{p_1}{p_0}=z_1-z_0.
$$
This connects the binary score difference with the <logit> link function. It does not make every coordinate of a multiclass score vector a separately defined binary log-odds.
A common shift also differs from rescaling. Dividing $(2,0)$ by two gives $(1,0)$ and first-class probability $e/(e+1)\simeq0.7311$. This is the transformation used in <temperature scaling> with temperature two. Positive scaling preserves the selected class within an example while changing confidence.<On Calibration of Modern Neural Networks by Chuan Guo and others>[§4.2, Eq. (9), temperature scaling]{ref}
When interpreting a <neural network logit>, state its score vector and probability transformation. Comparing raw coordinates from independently shifted parameterizations can otherwise suggest a confidence change where the actual probability vector is identical.
Back to article page
