Supervised learning Created 2026-10-14 Updated 2026-10-20
Supervised learning fits a prediction rule from training data containing both inputs and associated target responses. Regression analysis predicts numerical responses; classification in statistical learning predicts class labels. A loss function specifies which prediction errors matter, and generalization concerns performance beyond the observations used to fit the rule.
Test set Created 2026-09-28 Updated 2026-10-14
A test set is held out from both parameter fitting and model selection and is used to estimate the final selected procedure's prediction error. A validation set influences the choice; a test set assesses that choice after it is fixed. The distinction concerns how observations are used, not their file names.
For a fixed rule and an independent test set of size from the target probability distribution,This test error is an estimate, with sampling uncertainty; it is not a guarantee for each future prediction. Dataset shift changes what performance the estimate represents.
In the overfitting example, all degree choices are fixed in advance, cross-validation uses only training data, and the selected degree is locked before evaluating 400 separate points. Displaying the other curves afterward explains their behavior; it does not authorize selecting by those scores. Repeatedly inspecting a test set and changing the model makes it part of model selection, introducing data leakage and potentially optimistic assessment. [ref]
https://jmlr.org/papers/v11/cawley10a.html. Cawley and Talbot, over-fitting in model selection, 2010
Training epoch 2026-11-05
A training epoch is conventionally one pass through the training data used by an iterative learning procedure. Several parameter updates can occur within that pass. A mini-batch supplies a group of examples for an update, while the epoch counts progress through the data collection rather than counting individual updates.[ref]
Optimizing Model Parameters in the PyTorch tutorials. Hyperparameters, number of epochs and batch size; Optimization Loop, outer epoch and training loop
Suppose there are training examples. Visit each exactly once per pass, form mini-batches of at most examples and retain the smaller final group. Their sizes are . With one update for every mini-batch, an epoch hasupdates. Five epochs therefore make fifteen updates and fifty example visits. Fifty visits do not mean fifty distinct examples: the same ten examples are reused.
Changing the mini-batch size changes the update count. Taking all ten examples together gives one update per epoch under the same rule, while groups of two give five. An epoch count alone consequently does not specify the number of stochastic gradient descent steps, the amount of computation or the final fitted model.
The finite-pass assumption matters. Dropping an incomplete final group in this example gives two updates and only eight visits during that iterator traversal. Reordering the examples on later passes can change which examples are omitted. Sampling with replacement can repeat some examples before visiting others, so a chosen number of sampled updates need not be a literal once-per-example pass. Report those conventions rather than assuming that every loop named epoch guarantees full coverage.
The learning rate schedule needs a declared clock too: changing the learning rate once per epoch differs from changing it once per mini-batch. The optimization tutorial separates the outer loop from the operations that clear gradients, perform backpropagation and update parameters inside the training loop.[ref]
Optimizing Model Parameters in the PyTorch tutorials. Optimizer, zero_grad/backward/step; Full Implementation, nested training and epoch loops
Keeping data passes and update steps distinct makes training runs easier to compare. Training error Created 2026-09-24 Updated 2026-10-14
Training error is the average loss function evaluated on the observations used to fit the prediction rule. It is the empirical risk of the fitted rule on its own training data. With squared-error loss, it is the average squared regression residual, measuring fit to observed responses rather than performance on fresh responses. Flexible fitting usually makes it optimistically biased for prediction error, as quantified by prediction optimism.
For a reproducible polynomial regression example, take , , and the known noiseless target . Construct responses with fixed disturbancesOrdinary least squares over polynomials of polynomial degree at most 1, 2, and 7 gives training errors , , and . The monotone improvement follows because these candidate families are nested: every smaller-degree fit remains available in the larger family. This argument concerns minimum training error, not generalization, and assumes the fitting mathematical optimization is solved. Degrees 2 and 7 fitted instead to the exact noiseless responses have mathematically zero training error; the script obtains approximately and from rounding. Extra flexibility alone does not force overfitting.
Two-spin Boltzmann learning 2026-10-18
Take two visible bipolar units, no hidden units, zero thresholds, , and . As original training data, assign probabilities each to and and each to and . The data pair product averages to . Enumeration of all four states gives two weights and two , hence the partition function andStarting at , the positive-minus-negative difference is . A gradient ascent step of size gives . Each aligned state now has probability and each opposed state , bringing the model toward the data. The log-likelihood increases from to . Solving gives , which reproduces the data exactly. Since , this is the unique maximum. These are exact enumerations and their decimal evaluations, with no sampling or laboratory measurements.
Exact probabilities and positive-minus-negative learning for two visible units
. Left: data and model probabilities before and after one gradient ascent step. Right: the data pair-product average and the model average determine the log-likelihood derivative; their intersection gives the optimal coupling. Details. Unsupervised neural network models of the ventral visual stream by Chengxu Zhuang and others Created 2026-10-24 Updated 2026-10-31
Source. Published 2021. Relevant locator: Abstract and Results; PNAS 118, e2014196118: contrastive learning, multi-area prediction and child-view training data.
Yoshua Bengio 2026-11-05
Yoshua Bengio coauthored a 2003 neural language model that learns word embeddings jointly with a probability model for word sequences. The aim is to share statistical information among related contexts rather than assign unrelated parameters to every observed sequence.[ref]
For example, a learned vector for a word can contribute to predictions in many contexts. Updating that vector through one training example can therefore affect predictions elsewhere, providing a mechanism for generalization. This is an explicit computational use of a distributed representation, not a claim that nearby vectors always encode the same linguistic relationship. Bengio also coauthored Deep Learning by Ian Goodfellow and others.
