 Label smoothing

Label smoothing replaces a sharply concentrated training target with a mixture that assigns some probability to alternative labels. For a $K$-class example whose observed label is $y$, a common uniform version usesHere $p$ is the predicted probability vector and $0\le \epsilon <1$ controls smoothing. The target still sums to one. This is a regularization of the training objective, not a statement that the example is known to have several simultaneously correct labels.^[ref]

With three classes and $\epsilon =0.1$, the observed class receives $0.9333\dots$ and each other class $0.0333\dots$. This convention differs from assigning $0.9$ to the observed class and splitting $0.1$ only among the incorrect classes. State the convention when reporting a smoothing parameter.

For a softmax function applied to logits $z_k$, the categorical cross-entropy loss has gradient $\partial L/\partial z_k=p_k-q_k$. With fixed smoothed targets and unconstrained probabilities, the loss is minimized at $p=q$. The three-class example consequently has optimum true-versus-other logit gap $\log (0.9333\dots /0.0333\dots )=\log 28\approx 3.332$, rather than requiring an unbounded gap to approach a one-hot target.^[ref]

The changed objective can discourage extreme training predictions. Its effect on generalization, calibration or another downstream task is an empirical question; the derivation alone does not guarantee improvement on every dataset. Evaluation should therefore retain the distinction between a training regularizer and the true, unknown distribution of labels.