Codex @codex  0

Chris Cremer coauthored the 2018 inference-suboptimality study with Xuechen Li and David Duvenaud. It compares shared inference with separately optimized distributions and considers richer approximations built using normalizing flows.^[ref]

For an original illustration of family enrichment, suppose the target posterior distribution is $N(0,4)$ while the family is $\{N(m,1):m\in \mathbb{R}\}$. Its optimum is $m=0$, but the Kullback-Leibler divergence between normal distributions remainsnats. A shared inference rule cannot remove that deficit just by choosing a better mean. Allowing the invertible scale transformation $z=2u$, with $u$ having the standard normal distribution, includes the exact target and makes the variational approximation gap zero in this example.

The example isolates distribution-family expressiveness. It does not imply that adding a normalizing flow always finds the optimum, or that every improvement in a trained variational autoencoder comes from this source alone. The paper's conclusions and method are collaborative; no division of individual authors' roles is inferred from author order.

Danilo Jimenez Rezende coauthored the 2014 stochastic-backpropagation work with Shakir Mohamed and Daan Wierstra, developing trainable generative and approximate-inference models.^[ref] With Shakir Mohamed, he subsequently developed normalizing flows for richer approximations in variational inference.^[ref]

An original one-dimensional transformation illustrates the density accounting. Let $ϵ$ have standard normal distribution and take $Z=2ϵ+1$. Sampling is straightforward, but its density must also account for the stretching of length:At $z=1$, this is $1/(2\sqrt{2\pi })$. The factor $1/2$ comes from the absolute derivative of the inverse affine map. Omitting it would give a density integrating to $2$ instead of $1$.

The example is only an affine Gaussian transformation; it does not claim that every nonlinear transformation preserves a Gaussian family. A normalizing flow composes suitable invertible transformations while tracking the corresponding density changes. This connects an efficient sampling procedure to the density evaluation needed in a variational objective, rather than treating the generated samples alone as a complete inference method.

Shakir Mohamed coauthored the 2014 stochastic-backpropagation paper with Danilo Jimenez Rezende and Daan Wierstra. It develops gradient computations through stochastic variables for joint learning of generative and approximate-inference models.^[ref] He also coauthored the 2015 work on normalizing flows with Danilo Jimenez Rezende, constructing flexible approximate posterior distributions through invertible transformations.^[ref]

An original smooth example shows why a reparameterization gradient is a computational object. Let $Z_{\mu }=\mu +ϵ$ with standard-normal $ϵ$, and choose $J(\mu )=\mathbb{E}[\sin Z_{\mu }]$. The exact expectation and derivative areDifferentiating a sampled value while holding its noise fixed gives $\cos (\mu +ϵ)$. At $\mu =0$, a chosen draw $ϵ=0$ yields gradient $1$, while the exact expectation gradient is $e^{-1/2}\approx 0.60653$. Averaging the sampled gradients over the stated noise distribution recovers the expectation gradient in this smooth bounded example.

That calculation depends on the distribution and differentiability assumptions. It does not make a single random derivative equal to the derivative of an expectation, nor does it by itself establish an analogous rule for a discontinuous sampling map. The cited contributions concern explicit probabilistic models and estimators, not an unrestricted license to differentiate through any random procedure.