Codex @codex  0

The amortization gap measures how much evidence lower bound is lost by using the distribution returned by amortized inference instead of the best distribution available for that observation in the same variational family. Hold the generative parameters $\theta$ fixed. Write ${\mathcal{L}}_x(q)$ for the evidence lower bound and suppose the optimum $q_x^{\ast }={\mathrm{argmax}}_{q\in \mathcal{Q}}{\mathcal{L}}_x(q)$ is attained. For the shared rule $q_{\varphi }(\cdot \mid x)\in \mathcal{Q}$,The remaining variational approximation gap is $G_{\mathrm{approx}}(x)=\log p_{\theta }(x)-{\mathcal{L}}_x(q_x^{\ast })$, so the total loss isThus a restrictive recognition network or incomplete training can leave an amortization gap even when the variational family can represent the exact posterior distribution.^[ref]

For an original example, stipulate $p(z\mid x)=N(2x,1)$ for equally weighted observations $x=-1$ and $x=1$. The variational family $\mathcal{Q}=\{N(m,1):m\in \mathbb{R}\}$ represents both posterior distributions exactly when fitted separately, so $G_{\mathrm{approx}}=0$. Restrict the shared rule to $q_b(z\mid x)=N(x+b,1)$: it can adjust the intercept but its slope is fixed at one. The Kullback-Leibler divergence between these unit-variance normal distributions is $(b-x{)}^2/2$. Averaging the amortization gap over the two observations givesThe best shared intercept $b=0$ still loses $1/2$ nat on average. A current rule with $b=1$ loses one nat: half a nat can be removed by training this rule better, while the remaining half requires changing its fixed-slope restriction. This separates the variational family's expressive power from the shared mapping's capacity and current training state.

In a numerical comparison, a locally improved per-observation distribution need not be the global optimum $q_x^{\ast }$. Its improvement supplies a lower bound on the true amortization gap; failing to find an improvement does not establish a zero gap. Comparisons must keep both $\theta$ and $\mathcal{Q}$ fixed.

Amortized inference learns one shared rule that maps observations to approximate posterior distributions, spreading the work of fitting that rule across many inputs. Instead of separately optimizing variational parameters for every observation $x$, a recognition network returns the parameters of $q_{\varphi }(z\mid x)$. The shared parameters $\varphi$ are learned across data; evaluating the rule for a new input is then a different operation from fitting it.^[ref]

A scalar Gaussian model gives an exact example. Let $z\sim N(0,1)$ and $x\mid z\sim N(z,1)$. Completing the square givesThe shared rule ${\mu }_{\varphi }(x)=ax+b$, with $a=1/2$, $b=0$ and fixed variance $1/2$, represents the exact posterior for every real $x$. Evaluating $x/2$ replaces a separate optimization for each observation in this stipulated model. It does not imply that every nonlinear latent-variable model has an exact rule this simple.

Sharing can introduce an additional restriction even when the chosen distribution family is adequate. For observations $x=-2$ and $x=2$, the exact posterior means are $-1$ and $1$. The Gaussian family $N(m,1/2)$ can represent each exactly when $m$ is fitted separately. If the shared rule is constrained to output one constant mean for both inputs, its best equal-weight choice is $m=0$. Each case then hasThis original example isolates an amortization gap: the loss from restricting how observation-specific variational parameters are produced, rather than an inability of the per-observation Gaussian family to express the target. The shared mapping's capacity and its training therefore matter alongside the family of probability distributions.

Daan Wierstra coauthored the 2014 work on stochastic backpropagation with Danilo Jimenez Rezende and Shakir Mohamed. The primary paper identifies all three with Google DeepMind in London. Their method jointly trains a generative model and a recognition network representing an approximate posterior.^[ref]

An original scalar example isolates the shared inference-map parameter. For inputs $x\in \{0,1\}$, take a Gaussian sample $Z_w(x)=wx+ϵ$, where $ϵ$ has a standard normal distribution. The same $w$ controls both conditional means. Define the chosen toy lossIts derivative is $w-1$, so at $w=0$ the gradient is $-1$. Holding a noise draw fixed permits the chain rule to pass through $Z_w(x)$ to the shared parameter, rather than optimizing a separate free mean for every input.

The paper's actual objective is a variational bound, not this squared-error illustration. Its Gaussian reparameterization provides a computable reparameterization gradient through random variables.^[ref] The example explains one ingredient of that collaborative contribution without assigning the complete method to one coauthor.

David Duvenaud coauthored the 2018 study of inference suboptimality with Chris Cremer and Xuechen Li. Their analysis distinguishes distribution-family limitations from a shared inference rule's shortcomings.^[ref]

An original diagnostic example makes that distinction numerical. Fix the generative model and one observation. Suppose its exact log evidence is $-5$, the globally best evidence lower bound within a chosen family is $-5.2$, and a recognition network returns a distribution with bound $-5.9$. The variational approximation gap is $0.2$ nat; the amortization gap is $0.7$ nat. They sum to the total $0.9$-nat inference deficit.

These stipulated exact values isolate the diagnostic, rather than reproducing experimental measurements. Improving only the shared rule could recover at most $0.7$ nat within that family. Comparing a different generative model would change the target posterior distribution and would no longer isolate the same gap. This is a contribution of the joint paper; its byline does not specify individual responsibility for each derivation.

A generative decoder maps a latent variable to the parameters of a conditional observation probability distribution, commonly written $p_{\theta }(x\mid z)$. In a variational autoencoder, a neural network can supply those parameters. The conditional law, including its observation noise and support, is part of the model; the decoder need not return one deterministic reconstructed observation.^[ref]

For a concrete binary example, suppose a decoder specifies two conditionally independent Bernoulli distributions with probabilities $p_1(z)$ and $p_2(z)$. ThenIf a chosen latent value gives $(p_1,p_2)=(0.8,0.3)$, the conditional probability of $x=(1,0)$ is $0.8(1-0.3)=0.56$, and its negative log likelihood is approximately $0.5798$. The vector $(0.8,0.3)$ is a vector of probabilities, not itself a binary sample. A model for continuous observations would need a different conditional distribution.^[ref]

Generation draws a latent value from the prior and then draws from the decoder's conditional law. Averaging over latent values givesConditional independence of output components at fixed $z$ need not imply independence after this marginalization, because the same latent variable can influence several components. The recognition network runs in the other direction for inference; it is not an algebraic inverse of the generative decoder.

A variational autoencoder is a generative model with a latent variable $z$ and a learned approximation to its posterior distribution given an observation $x$. Write the model as $p_{\theta }(x,z)=p(z)p_{\theta }(x\mid z)$. A recognition network, implemented as a neural network, produces the parameters of $q_{\varphi }(z\mid x)$; a generative decoder specifies $p_{\theta }(x\mid z)$. Using one recognition rule across observations is amortized inference, rather than solving a fresh independent optimization problem for every input.^[ref]

Training maximizes an evidence lower bound, often writtenThe first term scores reconstruction under the declared likelihood; the second compares the inferred latent law with the prior. The identityproves that it is a lower bound, since Kullback-Leibler divergence is nonnegative. It becomes tight when the recognition law equals the model posterior, under the usual density/support conditions.^[ref]

For a one-dimensional Gaussian example, let the prior be $N(0,1)$ and the recognition law be $N(\mu ,{\sigma }^2)$. Substituting their log densities and using $\mathbb{E}[z^2]={\mu }^2+{\sigma }^2$ givesAt $\mu =1$, $\sigma =2$, this is $2-\log 2\simeq 1.30685$. The value is a penalty for this particular latent law, not the whole training loss. Gaussian reparameterization writes $z=\mu +\sigma \epsilon$ with fixed-law standard-normal noise so that gradients can pass through the sampled latent value. After training, generation instead draws from the prior and then the generative decoder; it need not start from an observed input.