= Ilya Loshchilov
{c}
<Ilya Loshchilov> is a researcher in <optimization> and <machine learning> whose work with <Frank Hutter> includes <SGDR> and <AdamW>. His Freiburg institutional profile identifies the same author and lists the two papers; it is an alumni record, rather than evidence of a current appointment.<Ilya Loshchilov profile at the Freiburg Machine Learning Lab>[Named alumni profile and 2017/2019 publication entries]{ref}
Their 2017 <SGDR> paper introduced a <learning rate schedule> that decreases the rate along a <cosine> curve, then raises it at a warm restart while retaining the current parameters. In one cycle,
$$
\eta(u)=\eta_{\min}+\frac{\eta_{\max}-\eta_{\min}}2\left(1+\cos\frac{\pi u}{T}\right).
$$
Here $u$ is elapsed cycle progress and $T$ its length. With $\eta_{\min}=0$ and $\eta_{\max}=0.1$, the rates at $u=0,T/2,T$ are $0.1,0.05,0$. Raising the rate starts another exploration from the learned parameters; it does not reinitialize the model.<SGDR: Stochastic Gradient Descent with Warm Restarts by Ilya Loshchilov and Frank Hutter>[Section 3, equation 5 and preceding restart description]{ref}
Their 2019 <AdamW> work separates <decoupled weight decay> from the gradient-based update. For ordinary <gradient descent>, adding $\lambda\lVert\theta\rVert^2/2$ gives $\theta'=(1-\eta\lambda)\theta-\eta g$. Coordinate-dependent scaling can act differently on the penalty contribution, so that equivalence does not extend unchanged to an <adaptive gradient method>.<Decoupled Weight Decay Regularization by Ilya Loshchilov and Frank Hutter>[Section 2, propositions 1–2]{ref} Writing the equation makes the contribution clearer than comparing optimizer names alone.
\Image[https://ml.informatik.uni-freiburg.de/wp-content/uploads/people/ilya.png]
{title=<Ilya Loshchilov>, portrait linked from his Freiburg Machine Learning Lab alumni profile}
{width=280}
Back to article page
