Past exam of the advanced topics in statistical machine learning course of the University of Oxford / 2025 / 1 / c

Use the factorization $p(\theta ,\mathcal{D})=p(\theta \mid \mathcal{D})p(\mathcal{D})$ inside the evidence lower bound:HenceThe marginal likelihood is independent of the variational parameter $\varphi$, so maximizing the evidence lower bound is equivalent to minimizing the Kullback-Leibler divergence. Nonnegativity of the Kullback-Leibler divergence also gives $\mathcal{L}(\varphi )\le \log p(\mathcal{D})$.

The independent coordinates make the logarithm of the probability density ratio a sum. For coordinate $i$, its contribution to the Kullback-Leibler divergence isbecause ${\mathbb{E}}_q{\theta }_i^2={\varphi }^2$. Thus, retaining all constants,Here ${\sigma }_1,{\sigma }_2>0$ and $\varphi >0$, so both normal densities are nondegenerate.

The printed minimization of the evidence lower bound has no solution. To distinguish that literal result from the usual variational inference optimum, put $S={\sigma }_1^{-2}+{\sigma }_2^{-2}>0$. The preceding Kullback-Leibler divergence satisfiesIts unique global minimum occurs at ${\varphi }^2=2/S$. Since $\mathcal{L}=\log p(\mathcal{D})-\mathcal{J}$, this is the unique global maximum of the evidence lower bound:The optimal common variance is the harmonic mean of the two posterior variances. However, $\mathcal{J}(\varphi )\to +\infty$ as either $\varphi \downarrow 0$ or $\varphi \to \infty$. ThereforeThe source PDF confirms the minimization wording; replacing it by maximization would give the finite value above, consistently with part (i).