= Label smoothing
\b[Label smoothing] replaces a sharply concentrated training target with a mixture that assigns some probability to alternative labels. For a $K$-class example whose observed label is $y$, a common uniform version uses
$$
q_k=(1-\varepsilon)\mathbf1_{k=y}+\frac{\varepsilon}{K},\qquad
L=-\sum_{k=1}^K q_k\log p_k.
$$
Here $p$ is the predicted probability vector and $0\leq\varepsilon<1$ controls smoothing. The target still sums to one. This is a <regularization> of the training objective, not a statement that the example is known to have several simultaneously correct labels.<Rethinking the Inception Architecture for Computer Vision by Christian Szegedy and others>[Section 7: target mixture and uniform smoothing distribution.]{ref}
With three classes and $\varepsilon=0.1$, the observed class receives $0.9333\ldots$ and each other class $0.0333\ldots$. This convention differs from assigning $0.9$ to the observed class and splitting $0.1$ only among the incorrect classes. State the convention when reporting a smoothing parameter.
For a <softmax function> applied to logits $z_k$, the <categorical cross-entropy loss> has gradient $\partial L/\partial z_k=p_k-q_k$. With fixed smoothed targets and unconstrained probabilities, the loss is minimized at $p=q$. The three-class example consequently has optimum true-versus-other logit gap $\log(0.9333\ldots/0.0333\ldots)=\log28\approx3.332$, rather than requiring an unbounded gap to approach a one-hot target.<Rethinking the Inception Architecture for Computer Vision by Christian Szegedy and others>[Section 7: cross-entropy gradient and motivation concerning overconfident training predictions.]{ref}
The changed objective can discourage extreme training predictions. Its effect on generalization, calibration or another downstream task is an empirical question; the derivation alone does not guarantee improvement on every dataset. Evaluation should therefore retain the distinction between a training regularizer and the true, unknown distribution of labels.
Back to article page
