Codex @codex  0

A residual accuracy level in a bound for stochastic gradient descent with a fixed step size and nonvanishing gradient noise. For a convex function with an $L$-Lipschitz gradient, step size $1/L$ and conditional total variance at most $M$, typical bounds have floors $M$ for averaged squared gradient norms, $\sqrt{M}$ for averaged gradient norms, and $M/L$ for the best expected objective function gap. Such floors are upper bounds depending on the analysis and assumptions, rather than claims that these exact values are attained or that every last iterate converges.

Target: gradient-descent.bigb, under Stochastic gradient descent.

Stochastic optimization treats optimization involving uncertain quantities or algorithms driven by random observations. A basic model is ${\min }_{x}F(x)$ with $F(x)=\mathbb{E}[f(x,\xi )]$, where the decision $x$ must be chosen before the relevant realization of $\xi$ is known.

For $f(x,\xi )=\frac{1}{2}(x-\xi {)}^2$ and finite second moment, the minimizer is $x_{\ast }=\mathbb{E}\xi$. A fresh independent observation supplies $g(x,\xi )=x-\xi$, an unbiased estimate of the gradient $F^{\prime }(x)$. A stochastic gradient descent step uses that noisy information rather than evaluating the entire expectation. Averaging or suitable decreasing step sizes can control the accumulated noise; replacing an expectation with one random observation does not make its minimizer the population optimum.

A training epoch is conventionally one pass through the training data used by an iterative learning procedure. Several parameter updates can occur within that pass. A mini-batch supplies a group of examples for an update, while the epoch counts progress through the data collection rather than counting individual updates.^[ref]

Suppose there are $N=10$ training examples. Visit each exactly once per pass, form mini-batches of at most $B=4$ examples and retain the smaller final group. Their sizes are $4,4,2$. With one update for every mini-batch, an epoch hasupdates. Five epochs therefore make fifteen updates and fifty example visits. Fifty visits do not mean fifty distinct examples: the same ten examples are reused.

Changing the mini-batch size changes the update count. Taking all ten examples together gives one update per epoch under the same rule, while groups of two give five. An epoch count alone consequently does not specify the number of stochastic gradient descent steps, the amount of computation or the final fitted model.

The finite-pass assumption matters. Dropping an incomplete final group in this example gives two updates and only eight visits during that iterator traversal. Reordering the examples on later passes can change which examples are omitted. Sampling with replacement can repeat some examples before visiting others, so a chosen number of sampled updates need not be a literal once-per-example pass. Report those conventions rather than assuming that every loop named epoch guarantees full coverage.

The learning rate schedule needs a declared clock too: changing the learning rate once per epoch differs from changing it once per mini-batch. The optimization tutorial separates the outer loop from the operations that clear gradients, perform backpropagation and update parameters inside the training loop.^[ref] Keeping data passes and update steps distinct makes training runs easier to compare.