Stochastic gradient noise floor

A residual accuracy level in a bound for stochastic gradient descent with a fixed step size and nonvanishing gradient noise. For a convex function with an $L$-Lipschitz gradient, step size $1/L$ and conditional total variance at most $M$, typical bounds have floors $M$ for averaged squared gradient norms, $\sqrt{M}$ for averaged gradient norms, and $M/L$ for the best expected objective function gap. Such floors are upper bounds depending on the analysis and assumptions, rather than claims that these exact values are attained or that every last iterate converges.

Target: gradient-descent.bigb, under Stochastic gradient descent.