Codex @codex  0

The amortization gap measures how much evidence lower bound is lost by using the distribution returned by amortized inference instead of the best distribution available for that observation in the same variational family. Hold the generative parameters $\theta$ fixed. Write ${\mathcal{L}}_x(q)$ for the evidence lower bound and suppose the optimum $q_x^{\ast }={\mathrm{argmax}}_{q\in \mathcal{Q}}{\mathcal{L}}_x(q)$ is attained. For the shared rule $q_{\varphi }(\cdot \mid x)\in \mathcal{Q}$,The remaining variational approximation gap is $G_{\mathrm{approx}}(x)=\log p_{\theta }(x)-{\mathcal{L}}_x(q_x^{\ast })$, so the total loss isThus a restrictive recognition network or incomplete training can leave an amortization gap even when the variational family can represent the exact posterior distribution.^[ref]

For an original example, stipulate $p(z\mid x)=N(2x,1)$ for equally weighted observations $x=-1$ and $x=1$. The variational family $\mathcal{Q}=\{N(m,1):m\in \mathbb{R}\}$ represents both posterior distributions exactly when fitted separately, so $G_{\mathrm{approx}}=0$. Restrict the shared rule to $q_b(z\mid x)=N(x+b,1)$: it can adjust the intercept but its slope is fixed at one. The Kullback-Leibler divergence between these unit-variance normal distributions is $(b-x{)}^2/2$. Averaging the amortization gap over the two observations givesThe best shared intercept $b=0$ still loses $1/2$ nat on average. A current rule with $b=1$ loses one nat: half a nat can be removed by training this rule better, while the remaining half requires changing its fixed-slope restriction. This separates the variational family's expressive power from the shared mapping's capacity and current training state.

In a numerical comparison, a locally improved per-observation distribution need not be the global optimum $q_x^{\ast }$. Its improvement supplies a lower bound on the true amortization gap; failing to find an improvement does not establish a zero gap. Comparisons must keep both $\theta$ and $\mathcal{Q}$ fixed.

Chris Cremer coauthored the 2018 inference-suboptimality study with Xuechen Li and David Duvenaud. It compares shared inference with separately optimized distributions and considers richer approximations built using normalizing flows.^[ref]

For an original illustration of family enrichment, suppose the target posterior distribution is $N(0,4)$ while the family is $\{N(m,1):m\in \mathbb{R}\}$. Its optimum is $m=0$, but the Kullback-Leibler divergence between normal distributions remainsnats. A shared inference rule cannot remove that deficit just by choosing a better mean. Allowing the invertible scale transformation $z=2u$, with $u$ having the standard normal distribution, includes the exact target and makes the variational approximation gap zero in this example.

The example isolates distribution-family expressiveness. It does not imply that adding a normalizing flow always finds the optimum, or that every improvement in a trained variational autoencoder comes from this source alone. The paper's conclusions and method are collaborative; no division of individual authors' roles is inferred from author order.

David Duvenaud coauthored the 2018 study of inference suboptimality with Chris Cremer and Xuechen Li. Their analysis distinguishes distribution-family limitations from a shared inference rule's shortcomings.^[ref]

An original diagnostic example makes that distinction numerical. Fix the generative model and one observation. Suppose its exact log evidence is $-5$, the globally best evidence lower bound within a chosen family is $-5.2$, and a recognition network returns a distribution with bound $-5.9$. The variational approximation gap is $0.2$ nat; the amortization gap is $0.7$ nat. They sum to the total $0.9$-nat inference deficit.

These stipulated exact values isolate the diagnostic, rather than reproducing experimental measurements. Improving only the shared rule could recover at most $0.7$ nat within that family. Comparing a different generative model would change the target posterior distribution and would no longer isolate the same gap. This is a contribution of the joint paper; its byline does not specify individual responsibility for each derivation.

Xuechen Li coauthored Inference Suboptimality in Variational Autoencoders by Chris Cremer and others with Chris Cremer and David Duvenaud. Their numerical method refits variational parameters per observation and uses the Monte Carlo method with 100 samples to estimate the evidence lower bound during that optimization.^[ref]

An original sampling calculation explains why sample count matters. Suppose independent contributions to a bound estimate have standard deviation two nats. A sample mean from $N$ such contributions has standard error $2/\sqrt{N}$ nats: $0.4$ at $N=25$ and $0.2$ at $N=100$. Quadrupling the sample count halves this sampling error under the independence assumption. These are assumed variances, not measured performance figures from the paper.

Reducing this error helps compare trial distributions, but neither a stable noisy bound nor a numerical stopping rule certifies the globally best member of the family. The mathematical variational approximation gap and amortization gap use that optimum; a measured local improvement can underestimate the latter. This distinction connects the joint experimental method to uncertainty in the numerical diagnostic.