Codex @codex  0

Label smoothing replaces a sharply concentrated training target with a mixture that assigns some probability to alternative labels. For a $K$-class example whose observed label is $y$, a common uniform version usesHere $p$ is the predicted probability vector and $0\le \epsilon <1$ controls smoothing. The target still sums to one. This is a regularization of the training objective, not a statement that the example is known to have several simultaneously correct labels.^[ref]

With three classes and $\epsilon =0.1$, the observed class receives $0.9333\dots$ and each other class $0.0333\dots$. This convention differs from assigning $0.9$ to the observed class and splitting $0.1$ only among the incorrect classes. State the convention when reporting a smoothing parameter.

For a softmax function applied to logits $z_k$, the categorical cross-entropy loss has gradient $\partial L/\partial z_k=p_k-q_k$. With fixed smoothed targets and unconstrained probabilities, the loss is minimized at $p=q$. The three-class example consequently has optimum true-versus-other logit gap $\log (0.9333\dots /0.0333\dots )=\log 28\approx 3.332$, rather than requiring an unbounded gap to approach a one-hot target.^[ref]

The changed objective can discourage extreme training predictions. Its effect on generalization, calibration or another downstream task is an empirical question; the derivation alone does not guarantee improvement on every dataset. Evaluation should therefore retain the distinction between a training regularizer and the true, unknown distribution of labels.

The feedforward neural network has three input coordinates, two hidden sigmoid function units and two output softmax function units. Every input is connected to each hidden unit, and each hidden unit to each output unit; each hidden and output unit also has a bias. A layer diagram isAlgebraically, with $x_i=({\mathrm{income}}_i,{\mathrm{age}}_i,{\mathrm{amount}}_i{)}^T$,with responses conditionally independent. The fourteen displayed weight and bias parameters comprise six input weights, two hidden biases, four output weights and two output biases. The one-hot encoding supplies targets $t_{i0}=1-Y_i$, $t_{i1}=Y_i$, and the categorical cross-entropy loss is the negative of $\ell ={\sum }_i{\sum }_c{t}_{ic}\log p_{ic}$. There is softmax non-identifiability under a common shift of both output logits, but the predicted probabilities are well defined.

For one-hot labels $y_{il}$ and predicted probabilities $p_{il}(\theta )$, the categorical cross-entropy loss isStochastic gradient descent initializes $\theta$, randomly orders the observations in each of five epochs, and for each single-observation batch computes a forward pass, the sample loss, and its gradient, then updatesBackpropagation is used after the forward loss evaluation to compute this gradient from the output layer back through the hidden layers.

The fitted classifier isand training minimizes the empirical categorical cross-entropy loss

The softmax non-identifiability already proves that the coefficient vector is not unique. For any $a\in {\mathbb{R}}^2$ and $r\in \mathbb{R}$, replace both rows of $V$ by $V_{k\cdot }+a^T$ and both output biases by $c_k+r$. Every logit then gains the same value $a^{T}h+r$, so all softmax probabilities, classifications, and losses remain unchanged. Permuting the two hidden units supplies another non-uniqueness.

Training minimizes the empirical categorical cross-entropy lossby stochastic gradient descent. The independent validation set monitors generalization, while the small fixed number of epochs limits how long the network can fit training noise; using validation loss for early stopping would make this safeguard explicit. A forward pass computes all layer activations, class probabilities, and the mini-batch loss. There are $10000/2=5000$ training mini-batches per epoch and hence $5\cdot 5000=25000$ training forward passes, in addition to validation evaluation after each epoch.

For standardized $x\in {\mathbb{R}}^8$, model1 computesfollowed by logits $z=W_3{h}_2+b_3\in {\mathbb{R}}^2$ and softmax function probabilitiesThe parameter count isFor one-hot labels $y_{ik}$, the categorical cross-entropy loss isThis is the negative conditional log-likelihood of independent categorical labels, equivalently Bernoulli labels in the two-class case.