Variational approximation gap ($G_{\mathrm{approx}}$)

The variational approximation gap is the part of the evidence lower bound deficit that remains even after optimizing separately for an observation within a specified variational family. With the generative model fixed and finite $\log p_{\theta }(x)$, defineThe equality uses the usual evidence lower bound identity for variational inference. A supremum allows for a best value that is not attained. The amortization gap separately measures failure of the shared inference rule to reach this value.^[ref]

An original example shows what independence can cost. Stipulate a centred two-dimensional multivariate normal distribution as the posterior distribution, with covariance matrixRestrict the variational family to centred independent normal distributions, $q=N(0,\mathrm{diag}(v_1,v_2))$ with $v_1,v_2>0$. This is a simple mean-field variational inference restriction. Writing $s=1-{\rho }^2$, the Kullback-Leibler divergence between normal distributions becomesEach variance contributes a term $v_i/s-\log v_i$, minimized at $v_i=s$. ConsequentlyFor $\rho =0.6$, the best independent variances are both $0.64$, and the unavoidable deficit is about $0.2231$ nat. Matching both marginal variances to one instead would give about $0.3394$ nat. The reverse Kullback-Leibler divergence therefore chooses narrower independent marginals here, while still failing to represent the correlation.

Enlarging the family to include the full covariance matrix permits $q=p$ and reduces this example's gap to zero. More generally, inclusion ${\mathcal{Q}}_1\subseteq {\mathcal{Q}}_2$ cannot increase the optimum approximation gap at a fixed model. Merely training the same shared rule longer cannot remove a strictly positive family restriction. Numerical improvements are not proofs that the infimum has been reached.