 Amortization gap

The amortization gap measures how much evidence lower bound is lost by using the distribution returned by amortized inference instead of the best distribution available for that observation in the same variational family. Hold the generative parameters $\theta$ fixed. Write ${\mathcal{L}}_x(q)$ for the evidence lower bound and suppose the optimum $q_x^{\ast }={\mathrm{argmax}}_{q\in \mathcal{Q}}{\mathcal{L}}_x(q)$ is attained. For the shared rule $q_{\varphi }(\cdot \mid x)\in \mathcal{Q}$,The remaining variational approximation gap is $G_{\mathrm{approx}}(x)=\log p_{\theta }(x)-{\mathcal{L}}_x(q_x^{\ast })$, so the total loss isThus a restrictive recognition network or incomplete training can leave an amortization gap even when the variational family can represent the exact posterior distribution.^[ref]

For an original example, stipulate $p(z\mid x)=N(2x,1)$ for equally weighted observations $x=-1$ and $x=1$. The variational family $\mathcal{Q}=\{N(m,1):m\in \mathbb{R}\}$ represents both posterior distributions exactly when fitted separately, so $G_{\mathrm{approx}}=0$. Restrict the shared rule to $q_b(z\mid x)=N(x+b,1)$: it can adjust the intercept but its slope is fixed at one. The Kullback-Leibler divergence between these unit-variance normal distributions is $(b-x{)}^2/2$. Averaging the amortization gap over the two observations givesThe best shared intercept $b=0$ still loses $1/2$ nat on average. A current rule with $b=1$ loses one nat: half a nat can be removed by training this rule better, while the remaining half requires changing its fixed-slope restriction. This separates the variational family's expressive power from the shared mapping's capacity and current training state.

In a numerical comparison, a locally improved per-observation distribution need not be the global optimum $q_x^{\ast }$. Its improvement supplies a lower bound on the true amortization gap; failing to find an improvement does not establish a zero gap. Comparisons must keep both $\theta$ and $\mathcal{Q}$ fixed.