= Durk Kingma
{c}
{tag=Mathematics author}
{wiki}
= Diederik P. Kingma
{c}
{synonym}
= Diederik Kingma
{c}
{synonym}
= Diederik Pieter Kingma
{c}
{synonym}
<Durk Kingma> is a <computer scientist> who publishes as <Diederik P. Kingma>. His personal profile describes research on <machine learning> and <generative models>, and records a 2017 doctorate at the <University of Amsterdam>.<Personal research homepage of Durk Kingma>[Brief Bio: formal name, nickname and2017 doctorate]{ref} The institutional thesis front matter gives his full formal name as Diederik Pieter Kingma.<Variational Inference & Deep Learning: A New Synthesis by Diederik Pieter Kingma>[Doctoral front matter]{ref}
\Image[https://dpkingma.com/files/portrait.jpg]
{title=Durk Kingma portrait linked from his personal research homepage}
{height=220}
Kingma and <Max Welling> coauthored <Auto-Encoding Variational Bayes by Diederik P. Kingma and Max Welling>. A <variational autoencoder> couples a <latent-variable model> with an approximating <posterior distribution> that can be optimized through <variational inference>. Their <reparameterization gradients> represent a simulated <random variable> as a differentiable transformation of parameter-independent noise, allowing derivatives through the transformation.<Auto-Encoding Variational Bayes by Diederik P. Kingma and Max Welling>[§§2.3–2.4, equations4–5 and Gaussian construction]{ref} Kingma also coauthored the <Adam> paper with <Jimmy Lei Ba>.<Adam: A Method for Stochastic Optimization by Diederik P. Kingma and Jimmy Lei Ba>[Full byline and Algorithm1]{ref}
An original scalar calculation makes the noise handling concrete. Use the <Gaussian reparameterization> $Z=\mu+\sigma\varepsilon$, with $\varepsilon$ having the <standard normal distribution> and $\sigma>0$. For $J(\mu)=\mathbb E[Z^2]=\mu^2+\sigma^2$, a one-draw <reparameterization gradient> with respect to $\mu$ is
$$
G_{\rm path}=2(\mu+\sigma\varepsilon),\qquad \mathbb E[G_{\rm path}]=2\mu,\qquad \operatorname{Var}(G_{\rm path})=4\sigma^2.
$$
The derivative holds the sampled $\varepsilon$ fixed. This <pathwise gradient> is justified here by direct calculation; a general interchange of derivative and <expected value> requires <differentiation under the integral sign> conditions.
Noise coupling matters even for a <central finite difference>. If the evaluations at $\mu+h$ and $\mu-h$ share the same $\varepsilon$, the difference of the two squares gives exactly $2(\mu+\sigma\varepsilon)$ for every $h>0$. This is a use of <common random numbers>. If instead the evaluations use independent <standard normal distributions> for their noise draws, the estimator
$$
G_{\rm ind}=\frac{(\mu+h+\sigma\varepsilon_+)^2-(\mu-h+\sigma\varepsilon_-)^2}{2h}
$$
still has <expected value> $2\mu$, but independence gives the <variance of Gaussian-square finite-difference gradients>
$$
\operatorname{Var}(G_{\rm ind})=\frac{\sigma^4+2\mu^2\sigma^2}{h^2}+2\sigma^2.
$$
Indeed, a <normal random variable> of mean $m$ and variance $\sigma^2$ has squared-value variance $2\sigma^4+4m^2\sigma^2$; sum the two independent squared-value variances and divide by $4h^2$. Thus shrinking $h$ need not suppress simulation noise.
For chosen $\mu=1$, $\sigma=2$ and $h=0.1$, both estimators have <expected value> 2, while their <variances> are \b[16 with shared noise and 2408 with independent noise]. The example is a separate exact quadratic model; its purpose is to explain the role of keeping base noise fixed, not to reproduce a published variational-autoencoder benchmark.
\Image[/gaussian-square-gradient-noise.png]
{title=Variance of independent-noise and shared-noise quadratic gradient estimates}
{description=Original analytic curves at mu = 1 and sigma = 2. Shared-noise central differences equal the pathwise derivative in this quadratic example; independent-noise variance grows as the finite-difference step shrinks.}
Back to article page
