= Jimmy Lei Ba
{c}
{tag=Mathematics author}
= Jimmy Ba
{c}
{synonym}
<Jimmy Lei Ba>, also known as Jimmy Ba, is a <computer scientist> whose published work includes <Adam> and <layer normalization>. His personal research biography identifies <Geoffrey Hinton> as his doctoral supervisor and describes a focus on efficient algorithms for <deep learning>.<Personal research homepage of Jimmy Ba>[Short bio and research description]{ref} The 2016 layer-normalization manuscript prints his <University of Toronto> affiliation and the name Jimmy Lei Ba.<Layer Normalization by Jimmy Lei Ba and others>[First-page full byline and affiliations]{ref}
\Image[https://jimmylba.github.io/assets/images/jimmy_ba.jpg]
{title=Jimmy Ba portrait linked from his personal research homepage}
{height=220}
Ba coauthored <Adam: A Method for Stochastic Optimization by Diederik P. Kingma and Jimmy Lei Ba> with <Diederik P. Kingma>. The algorithm maintains <optimizer state> based on <gradients>, so its update depends on the history of the optimization. <Adam bias correction> accounts for the early effect of initializing its exponential accumulators at zero.<Adam: A Method for Stochastic Optimization by Diederik P. Kingma and Jimmy Lei Ba>[Algorithm1 and §3, initialization bias]{ref}
The <layer normalization> paper, coauthored with <Jamie Ryan Kiros> and <Geoffrey E. Hinton>, addresses a different part of a <neural network>: it computes a <mean> and <variance> across feature coordinates of an individual example, then applies learned gain and bias. It avoids making those statistics depend on the other examples in the <mini-batch>.<Layer Normalization by Jimmy Lei Ba and others>[§3, equation3; Table1]{ref}
An original numerical example illustrates one precise symmetry. In the stabilized formula of <layer normalization>, set all gains to 1, all biases to 0, and <layer-normalization epsilon> to $\varepsilon=1/3$. For $z=(1,2,3)$, the coordinate <arithmetic mean> is 2 and the coordinate <variance> is $2/3$, so the normalized output is $(-1,0,1)$. Adding 10 to every coordinate gives $(11,12,13)$, whose output is again $(-1,0,1)$: <layer normalization is invariant under a common input shift> because the coordinate <mean> shifts by the same amount and the deviations remain unchanged.
The fixed <layer-normalization epsilon> affects scaling differently. With $2z=(2,4,6)$, the coordinate <variance> is $8/3$ and the output is $(-2/\sqrt3,0,2/\sqrt3)$, which differs from the original output. This calculation distinguishes the exact common-shift symmetry from idealized scale invariance when the stabilizer is omitted. It does not assign either numerical example to the paper's experiments.
Back to article page
