= Adam bias correction
{c}
{title2=$\widehat m_t=m_t/(1-\beta_1^t)$}
\b[Adam bias correction] compensates for the initial zero values in <Adam>'s exponentially weighted gradient histories. The basic algorithm uses
$$
\widehat m_t=\frac{m_t}{1-\beta_1^t},
\qquad
\widehat v_t=\frac{v_t}{1-\beta_2^t},
\qquad t\geq1.
$$
These factors normalize the coefficient sums of the two running histories.<Adam: A Method for Stochastic Optimization by Diederik P. Kingma and Jimmy Lei Ba>[Algorithm1, printed p2; section3, Initialization Bias Correction, printed p3]{ref}
For a scalar coordinate, suppose $m_0=0$ and
$$
m_t=\beta_1m_{t-1}+(1-\beta_1)g_t.
$$
Expanding the <recurrence relation> gives
$$
m_t=(1-\beta_1)\sum_{s=1}^{t}\beta_1^{t-s}g_s.
$$
If every $g_s$ has the same <expectation> $\mu$, linearity of expectation and the <geometric series> yield
$$
\mathbb E[m_t]
=(1-\beta_1)\mu\sum_{s=1}^{t}\beta_1^{t-s}
=(1-\beta_1^t)\mu.
$$
Thus $\widehat m_t$ is an <unbiased estimator> of $\mu$ under these assumptions. Correlation between different times is allowed by this argument. Applying the same calculation to squared gradients with constant raw <second moment> $\nu=\mathbb E[g_s^2]$ gives $\mathbb E[v_t]=(1-\beta_2^t)\nu$ and $\mathbb E[\widehat v_t]=\nu$.
For a constant supplied gradient of 2, choose $\beta_1=0.9$ and $\beta_2=0.99$. The first uncorrected histories are $m_1=0.2$ and $v_1=0.04$, while their corrected values are 2 and 4. The normalization removes the missing coefficient mass caused by initialization.
It does not make a changing history equal to the latest gradient. For an original deterministic example, let $\beta_1=1/2$, $g_1=1$ and $g_2=3$. Then $m_2=7/4$, so
$$
\widehat m_2=\frac{7/4}{3/4}=\frac73
=\frac13g_1+\frac23g_2,
$$
which differs from $g_2=3$. More generally, changing moment expectations produce a weighted average of past expectations. State the moment assumptions before interpreting “bias-corrected” as unbiased for a present distribution. The nonlinear parameter update built from the corrected histories is a separate object from either moment estimator.
Back to article page
