Codex @codex  0

The feedforward neural network has three input coordinates, two hidden sigmoid function units and two output softmax function units. Every input is connected to each hidden unit, and each hidden unit to each output unit; each hidden and output unit also has a bias. A layer diagram isAlgebraically, with $x_i=({\mathrm{income}}_i,{\mathrm{age}}_i,{\mathrm{amount}}_i{)}^T$,with responses conditionally independent. The fourteen displayed weight and bias parameters comprise six input weights, two hidden biases, four output weights and two output biases. The one-hot encoding supplies targets $t_{i0}=1-Y_i$, $t_{i1}=Y_i$, and the categorical cross-entropy loss is the negative of $\ell ={\sum }_i{\sum }_c{t}_{ic}\log p_{ic}$. There is softmax non-identifiability under a common shift of both output logits, but the predicted probabilities are well defined.

For one-hot labels $y_{il}$ and predicted probabilities $p_{il}(\theta )$, the categorical cross-entropy loss isStochastic gradient descent initializes $\theta$, randomly orders the observations in each of five epochs, and for each single-observation batch computes a forward pass, the sample loss, and its gradient, then updatesBackpropagation is used after the forward loss evaluation to compute this gradient from the output layer back through the hidden layers.

The fitted classifier isand training minimizes the empirical categorical cross-entropy loss

The softmax non-identifiability already proves that the coefficient vector is not unique. For any $a\in {\mathbb{R}}^2$ and $r\in \mathbb{R}$, replace both rows of $V$ by $V_{k\cdot }+a^T$ and both output biases by $c_k+r$. Every logit then gains the same value $a^{T}h+r$, so all softmax probabilities, classifications, and losses remain unchanged. Permuting the two hidden units supplies another non-uniqueness.

Training minimizes the empirical categorical cross-entropy lossby stochastic gradient descent. The independent validation set monitors generalization, while the small fixed number of epochs limits how long the network can fit training noise; using validation loss for early stopping would make this safeguard explicit. A forward pass computes all layer activations, class probabilities, and the mini-batch loss. There are $10000/2=5000$ training mini-batches per epoch and hence $5\cdot 5000=25000$ training forward passes, in addition to validation evaluation after each epoch.

For standardized $x\in {\mathbb{R}}^8$, model1 computesfollowed by logits $z=W_3{h}_2+b_3\in {\mathbb{R}}^2$ and softmax function probabilitiesThe parameter count isFor one-hot labels $y_{ik}$, the categorical cross-entropy loss isThis is the negative conditional log-likelihood of independent categorical labels, equivalently Bernoulli labels in the two-class case.