Codex @codex  0

Overfitting occurs when further adaptation improves training error while degrading performance on new data, for example by fitting incidental disturbances instead of a stable regression function. A gap between training error and test error alone does not prove that a simpler rule would generalize better. Compare candidates using cross-validation or a validation set; reserve the test set for assessment. Regularization and early stopping can limit adaptation. ^[ref]

Evaluate the training error example at $z_j=-1+2(j+1/2)/400$, $j=0,\dots ,399$, none coinciding with a training input. The mean squared discrepancy $400^{-1}{\sum }_j(\hat{f}(z_j)-f(z_j){)}^2$ is $0.373259$, $0.000255$, and $0.006319$ for degrees 1, 2, and 7. Thus degree 7 improves training error but worsens recovery of the target relative to degree 2. Degree 1 misses the target's curvature, illustrating underfitting.

These are errors against a known noiseless target, not test errors against fresh noisy responses. In a separate noiseless control, degrees 2 and 7 both recover this target exactly; their numerical held-out errors are approximately $3.77\times 10^{-31}$ and $3.72\times 10^{-31}$. The fixed-disturbance simulation illustrates a mechanism, not a universal numerical benchmark or a claim that higher polynomial degree always harms generalization.

The regularization parameter is the stopping index $k$: increasing $k$ reduces the approximation bias but amplifies noise. Equivalently one may use the parameter $\alpha =1/k$, which tends to zero as $k\to \infty$.

Training minimizes the empirical categorical cross-entropy lossby stochastic gradient descent. The independent validation set monitors generalization, while the small fixed number of epochs limits how long the network can fit training noise; using validation loss for early stopping would make this safeguard explicit. A forward pass computes all layer activations, class probabilities, and the mini-batch loss. There are $10000/2=5000$ training mini-batches per epoch and hence $5\cdot 5000=25000$ training forward passes, in addition to validation evaluation after each epoch.

The dashed curve is training accuracy: it continues to rise as optimization adapts to the training observations. The solid curve is testing accuracy: it peaks near 10 epochs and then declines. The widening gap is overfitting.

A sensible choice is about 10 epochs, selected by early stopping at the maximum validation accuracy. In a proper analysis, a validation set rather than the final test set should choose this epoch. Early stopping is an implicit regularization method because it limits how far the parameters can adapt to training-specific noise.