Neural network logit ($p_k=e^{z_k}/{\sum }_j{e}^{z_j}$)

A neural network logit is a real-valued class score produced before a probability transformation such as the softmax function. The vector of scores determines the class probabilities jointly; an isolated score is not itself a probability. Negative scores and scores larger than one are permitted.^[ref]

For scores $z_1,\dots ,z_K$, the softmax function givesAdding one common scalar $a$ to every coordinate leaves all probabilities unchanged:Consequently the absolute offset of a logit vector cannot be recovered from these probabilities.

For example, scores $(2,0)$ and $(7,5)$ both give first-class probability $e^2/(e^2+1)\simeq 0.8808$. The difference between the two scores, rather than the first coordinate alone, determines this binary probability. Indeed,This connects the binary score difference with the logit link function. It does not make every coordinate of a multiclass score vector a separately defined binary log-odds.

A common shift also differs from rescaling. Dividing $(2,0)$ by two gives $(1,0)$ and first-class probability $e/(e+1)\simeq 0.7311$. This is the transformation used in temperature scaling with temperature two. Positive scaling preserves the selected class within an example while changing confidence.^[ref]

When interpreting a neural network logit, state its score vector and probability transformation. Comparing raw coordinates from independently shifted parameterizations can otherwise suggest a confidence change where the actual probability vector is identical.