= Solution
For $K$ mutually exclusive classes, the output layer should provide $K$ nonnegative <probabilities> summing to one, with labels aligned consistently to output coordinates. A softmax activation on <logits> $a_k$ provides
$$
\boxed{p_k=\frac{e^{a_k}}{\sum_{\ell=1}^K e^{a_\ell}}},\qquad \sum_kp_k=1.
$$
Subtracting $\max_\ell a_\ell$ from all <logits> before exponentiation preserves these <probabilities> and prevents numerical overflow. Categorical <cross-entropy> $-\sum_ky_k\log p_k$ is appropriate for one-hot or probabilistic class targets. Independent sigmoid outputs fit a different setting in which multiple labels can be present simultaneously and their <probabilities> need not sum to one.
Back to article page
