Training epoch

A training epoch is conventionally one pass through the training data used by an iterative learning procedure. Several parameter updates can occur within that pass. A mini-batch supplies a group of examples for an update, while the epoch counts progress through the data collection rather than counting individual updates.^[ref]

Suppose there are $N=10$ training examples. Visit each exactly once per pass, form mini-batches of at most $B=4$ examples and retain the smaller final group. Their sizes are $4,4,2$. With one update for every mini-batch, an epoch hasupdates. Five epochs therefore make fifteen updates and fifty example visits. Fifty visits do not mean fifty distinct examples: the same ten examples are reused.

Changing the mini-batch size changes the update count. Taking all ten examples together gives one update per epoch under the same rule, while groups of two give five. An epoch count alone consequently does not specify the number of stochastic gradient descent steps, the amount of computation or the final fitted model.

The finite-pass assumption matters. Dropping an incomplete final group in this example gives two updates and only eight visits during that iterator traversal. Reordering the examples on later passes can change which examples are omitted. Sampling with replacement can repeat some examples before visiting others, so a chosen number of sampled updates need not be a literal once-per-example pass. Report those conventions rather than assuming that every loop named epoch guarantees full coverage.

The learning rate schedule needs a declared clock too: changing the learning rate once per epoch differs from changing it once per mini-batch. The optimization tutorial separates the outer loop from the operations that clear gradients, perform backpropagation and update parameters inside the training loop.^[ref] Keeping data passes and update steps distinct makes training runs easier to compare.