Past exam of the MSc in advanced computer science course of the University of Oxford / 2021 / Bayesian statistical probabilistic programming / 2 / d / Solution

The two limiting-case names in the printed bullets are reversed. Fully reparameterized continuous choices give a reparameterization gradient; non-reparameterized discrete choices give a score-function gradient estimator. Here are both reductions, including the extra term that vanishes only after taking an expectation.

When all choices are reparameterized with parameter-independent base noise, $s_{\lambda }=s$ and ${\nabla }_{\lambda }\log s=0$. The expression in part (c) becomesDifferentiating along the sampled transformation is precisely the pathwise gradient, also called the reparameterization gradient. There is no score function of the parameter-independent base law.

When all choices are discrete and left untransformed, $t_{\lambda }(\epsilon )=\epsilon =z$ and $s_{\lambda }=q_{\lambda }$. Set $S_{\lambda }(z)={\nabla }_{\lambda }\log q_{\lambda }(z)$ and $W_{\lambda }(z,x)=\log p(z,x)-\log q_{\lambda }(z)$. Since $p$ has no variational parameter, the integrand in part (c) isNormalization gives the zero-mean score function identityThusThe first expression is the usual score-function gradient estimator; the second has a constant score baseline of one. The two random integrands are generally different even though their expectations agree. A suitable constant score baseline can reduce variance, but one is not universally the optimal baseline.^[ref]

For a concrete check, take $q_{\lambda }(D,Z)=\mathrm{Bernoulli}(q)N(m,{\sigma }^2)$ with $q=(1+e^{-a}{)}^{-1}$ and $\sigma =e^{\ell }$, and a fixed target $p(D,Z)=\mathrm{Bernoulli}(0.4)N(D,1)$. Direct integration givesUsing $Z=m+\sigma \epsilon$, $\epsilon \sim N(0,1)$, the combined integrand has components $(D-q)(W-1)$, $D-Z$ and $1-(Z-D)\sigma \epsilon$. Their expectations are respectively $q(1-q)[\log (0.4/0.6)-\log (q/(1-q))+m-1/2]$, $q-m$ and $1-{\sigma }^2$, matching direct differentiation. Dropping the discrete score function term would incorrectly give zero for the first component. This calculation checks the combination without accepting the printed reversal.